Page tree
Skip to end of metadata
Go to start of metadata

Category:Application problem
Category:Deployment failed

Problem

Application's class loader leaks - "Error occurred while starting application" and "Unable to create folder"

Keywords

  • Error occurred while starting application
  • Unable to create folder
  • FSMakeFolder
  • Leaking class loader

Symptoms

Deployment returns warning due to the following errors: "Error occurred while starting application" and "Unable to create folder":

Error occurred while starting application sap.com/caf~core~ear and wait.

...

Reason: Clusterwide exception: server ID 603422350:com.sap.engine.
services.deploy.container.DeploymentException: Unable to create folder C
:\usr\sap\C42\JC60\j2ee\cluster\apps\sap.

...

Caused by: com.sap.engine.services.ejb3.container.ActionException:
Unable to create folder C:\usr\sap\C42\JC60\j2ee\cluster\apps\sap.
com\caf~core~ear\EJBContainer\applicationjars
#at com.sap.engine.services.ejb3.container.FSMakeFolder.

Solution

The problem is caused by Leaking class loader.

To determine what causes the problem follow the steps below:

1. Get the app name from the error message - in the sample above this is sap.com\caf~core~ear

2. Make sure the application is stopped. Use Telnet to check this. Go to the system using Telnet and type:

list_app

if the application is started, stop it by using the following command

stop_app

3. Check if the application's loader is leaking.
3.1. Open telnet and type
ll -leak

the command will list the leaking loader(s). If the application loaded is listed we have to ensure that it will not be cleared during GC. To do this we will force the GC.

3.2. Invoke GC - Start \usr\sap\SID\INSTANCE_ID\exe\run\sapjvm_5\bin -> jvmmon.exe and:

  • in the old jvmmon choose "1. Display VM list" and find out the VM on which the problem occurs. You can determine the VM by PID which is corresponding to the PID of cluster node in the Task Manager. Once you get the VmId (it might be 0) choose "3 Force full GC" and then choose "4 Force maximum compaction GC". Close jvmmon.exe. Now we are sure that all possible objects are GC-ed.
  • in the new jvmmon type "display vms" and find out the VM on which the problem occurs. You can determine the VM by PID which is corresponding to the PID of cluster node in the Task Manager. Once you get the VM id (it might be 0) type "force gc" and then type "force max gc", entering the VM id, when prompted. Close jvmmon.exe. Now we are sure that all possible objects are GC-ed.

3.3. In Telnet execute

ll -leak

and check if there are class loader leaks. The example below shows sample class loader leak.

>ll -leak
 Memory allocated: 85.68655%, Called System.gc(): true
 Suspicious leaked loaders (Please make sure that FullGC is triggered from GC.log)
 :
   [sap.com-caf_core~ear] unregistered on -> Thu Dec 14 09:34:15 EET 2006

If the application we are checking has a class loader leak - it will be listed.

4. If there is a class loader leak, follow the steps below in order to determine why:

4.1 Start \usr\sap\SID\INSTANCE_ID\exe\run\sapjvm_5\bin\jvmmon.exe and:

  • in the old jvmmon choose "9 Dump heap". The heap will be dumped into big file in \usr\sap\CID\JCXX\j2ee\cluster\server0 folder (there should be big file with similar name e.g. java_pid9924_1.hprof ).
  • in the new jvmmon type "dump heap". The heap will be dumped into big file in \usr\sap\CID\JCXX\j2ee\cluster\server0 folder (there should be big file with similar name e.g. java_pid9924_1.hprof ).

4.2 If you want to continue the investigation, use "Memory Analysis Tools" to analyze (version could be downloaded from HERE) and follow the instructions below. Or you can simply archive the heap dump file, obtained in the previous step, and send it to us, following the instructions for reporting.

4.2.1. Execute MemoryUsageDiggerGUI.bat to start it.
4.2.2. Load the the file with the created in the previous step with "Dump heap".
4.2.3. In menu section there is a button "Open in console". Click the button and in the console execute following command:

>LEAKING_LOADERS

There will be a similar result:

/>LEAKING_LOADERS
  1. Net Heap Size Retained Size Object Info
    --------------------------------------------------------------------------------
    
    1) 144 20 760 com.sap.vmc.core.impl.sapjvm.sharing.NativeSharedClassLoaderImpl @ 0x419cc918 sap.com#caf~core~ear
    --------------------------------------------------------------------------------
    
    144 20 760

4.2.4 If the class loader leaks - there should be a result similar to the sample in the previous point. Click on "0x419cc918" link and select "Open in Path from GC root" and then select "excluding weak references"

You will be able to see what causes the class loaded leak. See the sample above.

Sample logs

##1. Exception has been returned while the 'sap.
com/caf~core~ear' was starting. Warning/Exception :
Error occurred while starting application sap.com/caf~core~ear and wait.

Reason: Clusterwide exception: server ID 603422350:com.sap.engine.
services.deploy.container.DeploymentException: Unable to create folder C
:\usr\sap\C42\JC60\j2ee\cluster\apps\sap.
com\caf~core~ear\EJBContainer\applicationjars
#at com.sap.engine.services.ejb3.container.
ContainerInterfaceImpl$Actions.perform(ContainerInterfaceImpl.java:879)
#at com.sap.engine.services.ejb3.container.ContainerInterfaceImpl.
makeStartInitially(ContainerInterfaceImpl.java:757)

...

#at com.sap.engine.core.thread.execution.
CentralExecutor$SingleThread.run(CentralExecutor.java:168)
Caused by: com.sap.engine.services.ejb3.container.ActionException:
Unable to create folder C:\usr\sap\C42\JC60\j2ee\cluster\apps\sap.
com\caf~core~ear\EJBContainer\applicationjars
#at com.sap.engine.services.ejb3.container.FSMakeFolder.
perform(FSMakeFolder.java:48)
#at com.sap.engine.services.ejb3.container.CompositeAction.
perform(CompositeAction.java:81)
#at com.sap.engine.services.ejb3.container.
ContainerInterfaceImpl$Actions.perform(ContainerInterfaceImpl.java:873)

Related SAP Notes

N/A

  • No labels