See also SAP Note 1306843 for a reference on information to be provided when you open a support case.
Information to be provided for Support Messages
The following items are required for support messages:
- Version of software component LM-SERVICE on Solution Manager Java stack
- Version of component ST on Solution Manager ABAP stack
- http Remote connection (if not conflicting with company policy) to
- Solution Manager Java stack
- Solution Manager ABAP stack
- Introscope Webview
- credentials (in Secure Area) for a user with admin permission
- Log files from component with problems (EM, agent, ...)
- Screenshots of the problem (if this makes sense)
The mapping between Introscope versions and SAP versions of the Introscope delivery is as follows:
- Introscope follows a versioning scheme consisting of four parts: major.minor.servicepack.patchlevel, e.g. 10.5.2.91.
- SAP has a versioning scheme consisting of three parts: version SP supportpackage Patch patchlevel.
- The mapping is defined as follows for recent versions:
- Introscope major plus minor is mapped to SAP version
- Introscope servicepack is mapped to SAP supportpackage
- Introscope patchlevel is not mapped to SAP patchlevel but counted up stating from 0. Exact mapping you find in Introscope Release Notes.
- 10.5.2.27 maps to 10.5 SP2 Patch 0
- 10.5.2.91 maps to 10.5 SP2 Patch 1
- old Introscope 22.214.171.124 maps to 8 SP24 Patch 0 respectively 126.96.36.199 maps to 8 SP25 Patch 0
Enterprise Manager (EM)
Additional Information to be provided for EM Problems
- Description of the problem
- What kind of problem (slow Workstation / Webview, gaps in line chart, MoM losing connection to collectors, ...)?
- Screenshot of the problem (e.g. from Workstation / Webview screens)
- Reoccurrence: periodically, occasionally,
- Reproducible? If yes, how to reproduce?
- Time range
- When did the problem start occurring?
- Did the problem disappear without any action? If yes, when did it disappear?
- Thread dumps
- if you observe any kind of hang / crash situation trigger a series of thread dumps for the MoM at least, if possible also for the collectors. See below for details on triggering thread dumps.
- Log files
- Provide a complete set of log files (full contents of /logs folder) for MoM and all collectors. The log files must cover the time range when the problem occurred. Thread dumps will also arrive in the logs folder (em.log / EMService.log)
- Heap dumps
- In case of out of memory situations heap dumps may have been generated (java_*.hprof) in the EM installation folder. They might be needed as well
- Contents of the /data folder of the MoM installation
- This contains important self-monitoring metrics for post-mortem analysis
EM Additional Diagnostics Options
To enhance the diagnostics options the following configuration changes are recommended (at least in case problems occur)
- Run the Enterprise Manager on SAP JVM instead of the bundled Oracle / IBM JVM. This will provide more details in thread dumps, address some heap dump problems of Oracle JVM and enable a bunch of other diagnostics facilities. Make sure to use the same major Java version as present in the bundled JVM:
- Java 6 for EM <= 9.6,
- Java 7 for EM 9.7
- Java 8 for EM 10.1 and higher
- Oracle / SAP JVM: Set the JVM parameter -XX:+HeapDumpOnOutOfMemoryError
Enterprise Manager Log Files
Most relevant log files for problems with EM in sub directory logs are:
To get log out put with log level DEBUG in file
IntroscopeEnterpriseManager.log replace line
in file file <EM_Home>/config/IntroscopeEnterpriseManager.properties.
On Unix platforms the file
em.log in the EM home directory contains helpful information. All console output of the Java VM, including full thread dumps, goes here.
On Windows platforms the
logs folder contains an additional file
EMService.log. This is written by the Windows service wrapper and may contain important information in case the EM fails to start as Windows service.
Enterprise Manager Version
There are multiple ways to determine the EM version:
The log file
IntroscopeEnterpriseManager.logcontains the full version string at each startup of the EM. Example:
- Solution Manager discovers the version and displays it in Basic Configuration.
Adapt Enterprise Manager Java VM parameters
Depending on the OS there are different locations where you adapt the Java VM parameters.
Adapt the file <EM_HOME>/
bin/EMService.conf. The properties
wrapper.java.additional.<number> contain the Java parameters. Java heap size is specified in parameters wrapper.java.maxmemory and
Adapt the file
Introscope_Enterprise_Manager.lax in the EM home directory. Property
lax.nl.java.option.additional contains all JVM parameters. For Java heap size see Java parameter -Xmx and -Xms.
Recommended Java VM parameters for Enterprise Manager
In case the EM crashes with OutOfMemory (OOM) it makes sense to analyse the Garbage Collections (GC) in more detail. So setting additional java parameters
-verbose:gc -XX:+PrintGCTimeStamps -XX:+PrintGCDetails is recommended.
With java parameter -Xloggc:gc.log the GC output is redirected from file em.log to a dedicated file gc.log.
The parameter -XX:+HeapDumpOnOutOfMemoryError is usuful to trigger automatically a heap dump in case of OOM.
NOTE: heap dumps get a big as the Java heap size is configured. So make sure you have enough disk space available.
The bigger the Java heap size is configured the more it makes sense to work with parallel GCs by
setting additional java parameters -XX:+UseParNewGC -XX:+UseConcMarkSweepGC or -XX:+UseG1GC.
Start/Stop the Enterprise Manager
- Windows Service: Default Windows Service display name is "Introscope Enterprise Manager"
- Unix: use the script
- EMCtrl.sh start
- EMCtrl.sh stop
Register/Unregister Windows Service
The Windows Service to start/stop the Enterprise Manager usually gets registered by default during EM installation. In case you want to register or unregister this Windows Service manually you can do that with the script <EM_HOME>/bin/EMCtrl32.bat respectively EMCtrl64.bat. The command to register the service is EMCtrl64.bat register and EMCtrl64.bat unregister to remove the Windows Service.
The Windows Service display name and other details can be set in file <EM_HOME>\bin\EMService.conf. Changes to e.g. property wrapper.displayname modifying the default service display name "Introscope Enterprise Manager" of an altready registered Windows Serive only take effect after unregistering and registering the Windows Service.
Other properties you may need to change e.g. if you run several EMs on the same Windows host are wrapper.name and wrapper.description.
Collect Thread Dumps from the Enterprise Manager
If the Enterprise Manager appears stuck it may make sense to trigger a thread dump. The procedure differs depending on the operating system and the mode it is started.
If the EM is running as service (which is normal): Open a command prompt as adminsitrator and issue the command
sc control IScopeEM 255 (IScopeEM is the default name of the service). The thread dump will be appended to the service log file
logs/EMService.log. For processing in any thread dump analysis tools you will have to remove the wrapper prefix added to every line.
Example output of the command:
Using jmxremote or SAP JVM is an alternative, but requires extra configuration.
If the EM is launched interactively (Introscope_Enterprise_Manager.exe) you can hit Ctrl-Break to trigger a dump. Unfortunately you get it on the console, difficult to capture.
On OS level identify the process ID of the Enterprise Manager, e.g. via the command ps -ef|grep java.
Send signal 3 to this process:
kill -3 pid.
The thread dump will be written to the file
em.log in the EM installation directory
Collect Heap Dumps from the Enterprise Manager
If the EM crashes and you find errors like "JAVA heap space out of memory" in the IntroscopeEnterpriseManager.log may have either configured the EM with low Java heap size (then see "Adapt Enterprise Manager Java VM parameters" above to increase the heap size) or there could be a memory leak. For the latter heap dumps are needed for analysis. To get heap dumps in case of an OutOfMemory set the additional java parameter -XX:+HeapDumpOnOutOfMemoryError (see "Adapt Enterprise Manager Java VM parameters" above; a restart of the EM is needed). Setting this parameter automatically a heap dump is triggered if an OutOfMemory occurs. The heap dump could consist of more than one file and is usually named java_pid<pid>.*.
In case you want to trigger heap dumps manually set java parameter -XX:+HeapDumpOnCtrlBreak. With every triggered thread dump (see "Collect Thread Dumps from the Enterprise Manager") you then also get a heap dump.
NOTE: heap dumps get a big as the Java heap size is configured. So make sure you have enough disk space available.
Performance / Scalability / Capacity
The EM provides several self-monitoring metrics about its internal status. The metric
Overall Capacity (%) is also reported for each EM in the Solution Manager Administration work center. A value of 100% means that it has reached its capacity limit. Short peaks beyond 100% are not critical. Only if the capacity is permanently substantially beyond 100% you will experience problems like missing data points, poor performance, etc. You can find this
Overall Capacity (%) metric in the Introscope Investigator via this path: Superdomain -> Custom Metric Host -> Custom Metric Process -> Custom Metric Agent -> Enterprise Manager -> Overall Capacity (%).
A more detailed view is available in the Introscope Investigator tree via the following path: Superdomain -> Custom Metric Host -> Custom Metric Process -> Custom Metric Agent -> Enterprise Manager. If you select the node Enterprise Manager in the tree, the tab Overview on the right side displays eight charts on EM sanity, as shown in the screenshot. It reveals heap utilization, number of metrics and agents.
The EM collects Metrics about itself and reports them to a file called perflog.txt under 15 second time periods. By analyzing the entries in this file, it is possible identify beforehand if the EM is becoming overloaded and avoid future issues.
To check the EM health check the Introscope Troubleshooting WIKI page.
All known Webstart problems documented here were resolved with Introscope version 9.7 and subsequent versions. It might be easier to upgrade the Introscope Enterprise Manager and get a Webstart Workstation that works out of the box than to tweak the webstart for older versions. See SAP Notes 2534316 and 2285189.
The Java WebStart for the Introscope Workstation fails for higher patch levels of Java 7 and for Java 8 due to changes in the security checks of Java WebStart. In some cases the launch of the Workstation fails silently, in other cases an exception is shown.
In any case the following generic work-arounds exist:
- Use a Workstation that is explicitly installed on the client PC Note that Enterprise Manager and Workstation version must match
- Use the HTML-based Webview via a URL like http://xxxxx.yyy.corp:8081/webview
In addition, for the following Java patch levels work-arounds are available:
- Java 7 Update 45 (Java 1.7.0_45) and newer: Introscope Workstation 188.8.131.52 and older fail to start due to security related changes. Please see SAP Note 1565954. No action needed for Introscope 9.5.6
- Java 7 Update 51 (Java 1.7.0_51) and newer: Additional security changes get introduced causing the Workstation to fail again. As Workaround launch the Java Control Panel (Start->All Programs->Java->Configure Java). On the Security tab leave the Security Level as “High” and add the WebStart URL to the “Exception Site List”
Example for WebStart URL: http://xxxxx.yyy.corp:8081
- Java 7 Update 71 (Java 1.7.0_71) and newer: Workstation WebStart client fails again. Workaround for Introscope 9.5.6: add line <property name="jnlp.eclipse.security" value="osgi"/> to file <EM_HOME>/product/enterprisemanager/plugins/com.wily.introscope.workstation.webstart_<version>/WebContent/jnlp/workstation.jsp. The only work-around for Introscope 9.1.5 is to explicitly use Java 6, as described below in Section "Webstart Additional Hints".
Workstation Webstart: Expired Certificates as of March 1, 2015
Furthermore as of March 1, 2015, the CA APM Java JRE certificate has expired. This will impact all the customers who use web start client, version 9.7 and lower. Customers who initiate the workstation through the downloaded JNLP will be blocked if their security is set beyond medium. This issue is independent from issues related to individual Java Updates as listed above.
Below are options to work around this issue:
- Option 1: If the security level is set to high or above, the level should be brought down to medium. Once the security level is set to medium, the web start client can be initiated to start the workstation.
In order to do this -> Go to Control Panel -> Java - > Security and adjust the security level.
- Option 2: Add the web start URL to the exception site list. Once this site is added reinitiating the web start client starts the workstation.
In order to do this - > Go to Control Panel -> Java - > Security -> Click on Edit Site List and add the URL in the location box.
Webstart Additional Hints
We have not found a way to launch the Workstation 9.1.5 with recent Java 7 and Java 8 versions. You can, however, have multiple JRE versions installed on the same host. In particular, you could have Java 6 installed to launch the Workstation, and at the same time use a recent Java 8 version for webstart of other applications.
- Use an absolute path like
"C:\Program Files\Java\jre6\bin\javacpl.exe"to launch the Java Control Panel. Disable other Java versions in tab "Java".
Use an absolute path like
"C:\Program Files\Java\jre6\bin\javaws"http://emhost:8081/workstation to launch the Workststion.
- Use an absolute path like
- In some cases having multiple JRE/JDK versions installed might cause problems. Consider uninstalling JRE versions that you do not really need.
Try launching the Webstart bypassing the browser In a command window. This isolates any potential browser add-in issues and allows to use webstart even if the browser add-in is disabled: Use a call like this:
SAP / Introscope Push Service, Grey Metrics in System Monitoring
See separate page: Introscope Push
Missing Data in Early Watch Alert, Grey EWA reports
Follow Knowledge Base Article 1824881 to classify if the missing data should be provided via Introscope.
If you come to the insight that missing metrics indeed should come from Introscope then check KBA 2921337 for troubleshooting.
Java Bytecode Agent (BCA)
Introscope agents of a certain version can connect to Introscope Enterprise Managers of the same or higher version. It is not possible to connect them to an EM with a lower version than the agent is. For more information, refer to the Introscope JAVA Agent WIKI.
Relevant log files for problems with the BCA are:
- from directory
- from directory
- from directory
/usr/sap/SID>/<inst>/workof the managed system if it's a Netweaver java node:
dev_server0(contains the actually used JVM parameters)
If requested by SAP Development set in IntroscopeAgent.profile the property log4j.logger.IntroscopeAgent from
A restart is not required. Do not forget to reset the value back to INFO after investigating the issue.
The agent version can be determined via one of the following procedures:
- Check the log file
IntroscopeAgent*.logfor the right agent. The agent version is dumped at every restart of the Java VM.
- Check the Introscope Investigator tree. For each agent, the node Agent Stats contains
- Agent build and release
- SAP build timestamp
- location of the agent profile that is used
Byte Code Adapter does not start
Missing CPU Metrics
CPU node in the Investigator tree of the byte code instrumentation agent (e.g. Netweaver agent) is populated via a native library, the so-called platform monitor. There are multiple possible reasons why this node might be missing:
- there is no platform monitor extension for all supported Netweaver platforms. As fallback you can use the OS metrics as provided by the Introscope host adapter (under node SAP Host Agent)
- on Windows: If the user running the Java VM is not in the administrators group it must be added to the group
Performance Monitor Usersto obtain the privileges required to read performance data.
Missing JMX Metrics
For troubleshooting JMX metric collection in the agent you can increase logging of the JMX service by setting these two additional properties:
In general polling of particular MBeans will be switched off until the next restart of the Java VM as soon as any exception occurs for polling that MBean.
For Netweaver there is a known problem that reporting of http sessions ("Current Http Sessions") stops reporting. See SAP Note 2067705 for a fix.
Java Managed System does not start due to missing Agent.jar
If a Java Managed System has been instrumented for CA Introscope via the Java VM parameter
Agent.jar and the file
Agent.jar is not accessible by the instrumented Java VM then it fails to start. Possible reasons why the file
Agent.jar is not accessible:
- The corresponding Diagnostics Agent has been deleted before the CA Introscope instrumentation has been removed from the Java Managed System. The Java Managed System does not start anymore because the required
Agent.jarin the Diagnostics Agent file system has been deleted.
- The managed Java VM does not have read access to the file system tree where the Introscope agent is located (e.g. user SAPService<SID> on Windows starting the SAP Netweaver system)
In the first case the instructions for uninstalling the Introscope Byte Code Agent were not followed correctly.
You can find those instructions in the Introscope Installation or Setup Guide for your Solution Manager Release.
These installation or setup guides can be found at http://www.service.sap.com/instguides
=> Alphabetical Index => S
=> SAP Solution Manager
=> "your solution manager release"
=> "one of the sections on installations"
=> "respective Wily Introscope guide".
In these guides, please refer to the section titled "Uninstalling Introscope Agents". It is recommended to use the SAP Solution Manager provided mechanism to remove the problematic properties.
In case the Diagnostics Agent was already deleted, it may be necessary to delete the properties by hand. In case of problems please create a ticket at component SV-SMG-DIA-WLY-BCA.
Known problems for Bytecode Agent
Introscope Host Adapter (wilyhost)
Check if Introscope Host Adapter is connected to the Enterprise Manager
Missing ABAP metrics
Log Files and Log Levels
Relevant logfiles for problems with Introscope Host Adapter are:
SMDAgentApplication*.log(most content here)
SMDSystem*.log(in case wilyhost does not start up at all)
They are persisted in directory
The log level for
SMDAgentApplication.log might have to be increased to DEBUG for troubleshooting. This is done via diagnostics agent administration.
If the Introscope host adapter does not show up in the Investigator the file
jvm_SMDAgent.out may also be relevant. It corresponds to
IntroscopeAgent*.log of the BCA. To increase the log level for the Introscope runtime of the host adapter (in
jvm_SMDAgent.out) edit the application resource "IntroscopeSapAgent.profile" and change the property
Thread dumps for the Introscope host adapter means thread dumps for the whole diagnostics agent. They can be triggered in various ways, e.g.:
- Via the SAP Management Console / MMC: Right-Click on the java procees for the agent and click Dump Stack Trace
- On OS level (Unix): find out the process ID of the agent process and issue the command
kill -3 <pid>
In any case the thread dump is written to the file
std_smdagent.out in the
work directory of the diagnostics agent.