Skip to end of metadata
Go to start of metadata

The purpose of this wiki page is to provide details for troubleshooting the Managed System Configuration of Java systems.

1) Check the Workload Analysis Infrastructure

Refer to the Workload Analysis Infrastructure troubleshooting WIKI and ensure that all resources required are correctly configured.

2) Check the Managed System Configuration

The Managed System Configuration must be performed completely for the target managed system and their results must be all green.

Recheck all steps of the Managed System Configuration for the target system and ensure that there are no issues. Sometimes the steps/activities are marked as 'Update Needed', repeat all steps marked as 'Update Needed'. 

For Java stack systems:

  • Step 'Assign Diagnostics Agents': Ensure that all Diagnostics Agents are connected to the Solution Manager.
  • Step 'Create Users': The user SM_COLL<Solution Manager SID> is used to collect data from the managed system. Ensure that user SM_COLL<Solution Manager SID> has no issues, e.g. it is not locked
  • Step ''Configure Automatically': All activities within this step must finish green, especially 'Database Extractor Setup',  'Extractors Setup', 'Introscope Host Adapter' and 'Byte Code Adapter'. After the 'Byte Code Adapter' setup, it is required to restart the Java server of the managed system to start the Byte Code Adapter (Introscope Agent)
  • Step Configure Manually: Execute all mandatory steps.

3) Check the Data collected by the Introscope Host Adapter and Byte Code Adapter (Introscope Java Agent) 

To check all configured agent in the Introscope Investigator for each application instance of the managed system, start the Introscope Web View or use the Introscope Workstation.

  • Start Introscope Workstation --> New Investigator
  • Start Introscope Web View --> Investigator tab

Note: The data collected by the Introscope Host Adapters and Byte Code Adapters differ for each product version. Find examples for a certain types of systems below. The nodes available in the Introscope Enterprise Manager may differ depending on the product version of the managed system.

3.1) Expected Data in the Investigator for a Java Instance

When the managed system is Java only, the system needs to be monitored by Introscope Host Adapters and Byte Code Adapters (Introscope Agents). The Introscope Agents are necessary to monitor the applications running in the managed system JVM. As for ABAP systems, an entry for each application server of the managed system is expected in the Introscope Investigator. The difference in the Wily Investigator is that there will be one entry for the Introscope Host Adapters and one entry for the Byte Code Adapters running on the managed server.

The below example is for the system SLE, which has only one Java application server running on server 'vmw4371'.

  • By default, the WilyHost application connects to the Enterprise Manager using Process Name 'SAP HostAgent Process' and Agent Name 'SAP HostAgent <Diagnostics Agent Instance>'.  
  • By default, the Byte Code Adapter connects to the Enterprise Manager using Process Name 'SAP Netweaver' and Agent Name <SID_of_managed_system>_<instance>_<servernode>. The Byte Code Adapters are configured to monitor the Java server nodes, i.e. one entry per server node is expected. In this example, the managed system SLE only has one server node.

If the entries 'SAPHostAgent Process' and 'SAP HostAgent <Diagnostics Agent Instance>' are not available in the Introscope Investigator, the Introscope Host Adapter could not connect to the Enterprise Manager. In this case, check the file jvm_smdagent.log which is located in the work folder of the Diagnostics Agent. It provides information about the connection between the Introscope Host Adapter and the Enterprise Manager. If you use the Agents-on-the-fly feature, check the corresponding file jvm_smdagent_<logicalhostname>.log.

If the entries 'SAPHostAgent Process' and 'SAP HostAgent <Diagnostics Agent Instance>' are available, take a look in the data collected to have an overview about the data collected.

For all Java systems one must see a node with the System ID for the target managed system. The performance data of the Java instance is displayed under this node (SID of managed system). In the example above, the System ID is SLE and the instance name is vmw4371_SLE_00. If there are several managed systems running in the same host name, there should be one node for each managed system. If the node with the system ID is not available, the the Introscope Host Adapter could not start the action. This issue might happen due to several issues. In this cases, check the log SMDAgentApplication.log, this log will provide details about the issue.

If the entries 'SAP Netweaver' + <SID_of_managed_system>_<instance>_<servernode> are NOT available in the invstigator, this means that the Byte Code Adapter (ISAgent) could not connected to the Enterprise Manager or it is not running. In this case, check the IntroscopeAgent.log that is usually located in the temp folder of the Diagnostics Agent  \usr\sap\SMD\SMDA97\SMDAgent\temp. If you are using Diagnostics Agents on the fly and you selected the option "Deploy Byte Code Adapter in the Managed Instance path", the logs will be probably in the managed system path ((/usr/sap/<ManagedSID>/<InstanceID>/ByteCodeAdapter).

If the IntroscopeAgent.log does not exist or is not being updated for a long period, this means that the Byte Code Adapter (ISAgent) is not running. In this case, check the results of activity Byte Code Adapter Installation in step Configure Automatically of the Managed System Configuration for the target system. The activity Byte Code Adapter Installation is responsible to deploy the ISAgent files and configure the ISAgents parameters in the manged system. If the managed system runs on a UNIX server, refer to the WIKI page 'Byte Code Adapter does not start' this WIKI shows how to check this issue.

If the entries 'SAP Netweaver' + <SID_of_managed_system>_<instance>_<servernode> are available, then it is possible to have a look in the data collected to have an overview about the data collected.

The nodes displayed for the server nodes depends on the configuration performed during the Byte Code Adapter Installation in the Managed System Configuration. The performance data to be collected is configured by the Autoprobe Directive files there are selected during the Byte Code Adapter Installation. In the example above system SLE is a Portal system, this is why we see node 'SAP Portal'.

IMPORTANT: There are some known issues with ISAgent 8 and 9 which leads to missing metrics in the Workload Analysis. Refer to the following KBAs and check whether the ISAgent release that is being used is affected by the known issues:

1793553 - Few metrics are being reported for node 'SAP Netweaver' in wily due to error "/wily/core/config/ext could not be located. No extensions will be loaded."

1796399 - Few metrics are being reported for node 'SAP Netweaver' in wily due to error "AutoProbe will not run because the directives files could not be loaded ... hotdeploy"

2089621 - Few metrics being reported for node "SAP Netweaver" for PI systems

3.2) Host Data is Missing (CPU, DISKs) 

If the host data is missing from the workload analysis for EWA, e.g. CPU, Disks and Memory, it is required to check if the Introscope Host Adapter application is collecting the data. The host data is collected by the Introscope Host Adapter using the SAP Host Agent. The SAP Host Agent exposes some Web Services that are called by the Diagnostics Agent to get the host performance data. For all hosts, the SAPOscol node must be available in Wily:

If the SAPOscol node is not available for the target host, this means that the Diagnostics Agent could not connect to the SAP Host Agent. This issue can be caused by several reasons. Check section 'SAP Host Agent' of the outside discovery troubleshooting guide attached to note 1611483.

  • SAP Host Agent must have the version described in note 1365123.
  • The SAP host Agent porttypes are correctly configured as per note 1611483.
  • SAPHostAgent trusted connection must be enabled as per note 1611483.

To check the SAP host Agent web services for a host, it is possible to access the following URLs. These URLs will return the WSDL file.

  • http://<localhost>:1128/SAPOscol/?wsdl
  • http://<localhost>:1128/SAPHostControl/?wsdl

If the SAP Host Agent configuration is correct, check the SADAgentApplication.log of the affected host, this file will provide details about the issue.

5) Check Managed System Extractors

The performance data reported to the Enterprise Manager is collected by the Solution Manager system using the extractors. If the steps shows in section 3 are ok for the affected managed system, that is, the expected performance data is available for the manged system hosts/instances in Wily. Then one must check the extractors for the managed system to search for errors.

In Solman 7.10, the extractors can be found at:

  • Tx SM_WORKCENTER
  • SAP Solution Manager Administration
  • Infrastructure
  • Framework
  • Extractor Framework

In the Extractor Framework, search for the target manged system and check the status of ALL Workload Analysis extractors. They must have green lights. In tab 'Extractor Log', it is possible to see the number of collected records in collumn Records.


If you find errors in the workload analysis extractors for the managed system, check the EFWK Troubleshooting WIKI.

6) EarlyWatch Specific Checks 

The Early Watch Alert (EWA) for JAVA and Dual Stacks systems depends on the Workload Analysis data that is collected by the Diagnostics infrastructure. This section shows the troubleshooting steps of the most common issues with EWA reports.

Note: Before proceed with the specific check described below, pay attention in the following behaviors of how the EWA collects data from the Diagnostics BW infocubes:

  • The performance data must be available for the same period of the EWA document. So when checking the Workload Analysis for the affected system, ensure that you are checking the same period as the EWA document. Usually the period used is 'Last Week' in the Workload Analysis. The following picture describes where the period (Timeframe) can be selected in the Workload Analysis application.

  • The EWA required that the data is available in both granularities 'Hour' and 'Day'. In this case, it is required to ensure that the performance data is available in both granularities for the EWA period (Last Week). The following pictures describes how the granularities can be changed.

IMPORTANT: If the performance data can be found in granularity 'Hour' but not in granularity 'Day', then check whether the job E2E BI HOUSEKEEPING is correctly scheduled and running without issues in the SAP Solution Manager. The section 'Housekeeping for Standard BW Cubes' of the EFWK Setup WIKI provides more information about this job. If the job is not scheduled, run again activity 'Schedule Sol. Manager Background Jobs' in the 'Basic Configuration' (SOLMAN_SETUP) to schedule the job again.

6.1) Alert for Metric Type "APPLICATION THREADS" and Metric Name "ACTIVETHREADSCOUNT"

In the EWA the following alert is raised:

  • 'No performance data is returned from BW Infocube Infocube 0SMD_MPEH, metric type Application Threads metric name: ACTIVETHREADSCOUNT used in section Java System Data for <XXX>'

This error indicates that the J2EE Threads Usage is not being correctly collected for the affected system. It is possible to check if this data is being collected as follows:

  • Transaction SM_WORKCENTER
  • Root Cause Analsysis Workcenter
  • End-to-End Analysis
  • Start the Workload Analysis for the affected JAVA system
  • Check metric 'J2EE Thread Usage' under tab 'J2EE Engine'

If no data is being reported to this section, then it is required to check if the Byte Code Adapter (Introscope JAVA Agent) is collecting all the required information. Refer to section 'Expected data in the Investigator for a JAVA instance' of this WIKI and ensure that the Byte Code Adapter is reporting the performance data for the introscope.

If the Byte Code Adapter is reporting data correctly, then refer to section 'Check Managed System Extractors' of this WIKI and ensure that the Workload Analysis Extractors are running without issues.

Finally, also ensure that job E2E BI HOUSEKEEPING is correctly scheduled in the SAP Solution Manager. This job perform the aggregation of the BW information from the hour infocubes to the day infocubes. The EWA gets the information from the day infocubes, if this job is not running or has issues the EWAs are affected. Check section 'BI Housekeeping' of the EFWK Setup WIKI for more information about job E2E BI HOUSEKEEPING.

 6.2) Alert for Metric Type "Java Memory Usage" and Metric Name "% GC TIME (LAST 5 MINUTES)"

In the EWA the following alert is raised:

  • No performance data is returned from BW InfoCube. Infocube: 0SMD_MPEH , metric type: JAVA MEMORY USAGE , metric name: % GC TIME (LAST 5 MINUTES) used in section 'Java System Data for <XXX>'

This error indicates that the SAP GC data is not being correctly collected for the affected system. It is possible to check if this data is being collected as follows:

  • Transaction SM_WORKCENTER
  • Root Cause Analsysis Workcenter
  • End-to-End Analysis
  • Start the Workload Analysis for the affected JAVA system
  • Check metric 'Java Memory Usage' under tab 'J2EE Engine'

Solution Manager 7.20

The Solution Manager 7.20 collects the Java Memory Usage information which is reported by the Introscope JAVA Agent (Byte Code Adapter) under the VM node. Check in the Introscope Investigator where the metric % GC TIME (LAST 5 MINUTES) is being reported correctly as follows:

  1. Introscope Webview or Workstation;
  2. Investigator;
  3. Find affected system's server;
  4. Expand node SAP Netweaver until find VM node;
  5. Exapnd GC node;
  6. See metric named as 'GC Time (%) Last 5 Min'.

The following picture describes the metric in the Introscope Investigator:

If the entire node 'SAP Netweaver' is missing, check  Byte Code Adapter does not start WIKI for details about how to troubleshoot the startup of the Introscope JAVA Agent.

Known Issues:

  • 1) SAP Solution Manager 7.20 using Introscope Enterprise Manager 9.7: Wrong name for metric '% GC TIME (LAST 5 MINUTES)' causes issues in the data collection, as consequence the EWA has no % GC TIME (LAST 5 MINUTES) data.
    SOLUTION: KBA 2523393.
    2523393
    - Wrong name for metric '% GC TIME (LAST 5 MINUTES)' causes wrong Early Watch Alert for metric type: JAVA MEMORY USAGE

Solution Manager 7.10

In Solution Manager 7.10 the SAP GC data is collected by the Introscope Host Adapter (application of Diagnostics Agent). To troubleshoot the SAP GC information, it is required to check if the Action which collects the information is correctly started. For this, check in the Introscope Investigator if the SAP GC data is being collected. The steps are the following:

  • Access the Introscope Investigator using the Webview or Workstation
  • Find the affected server
  • Expand SAP Host Agent Process node under the affected server
  • Expand SAP Host Agent SMDA<XX> node
  • Check if the SAP GC node is available

The following pictures shows where to find the SAP GC node of the Introscope Host Adapter:

If the SAP GC node is not available, this means that the SAP GC Action could not be started. In this cases, the SMDAgentApplication.log of the affected Diagnostics Agent will show the root cause. An example where the SAP GC cannot be started is the following:

Solution: The SAP GC is collected from file std_server* which is located in the managed system work folder. In this case, it is required to ensure that the path to the files is correct. It is possible to compare the file shown in the log error or in the action definition in the SAPAgentConfig.xml.

The steps below shows how to find the SAPAgentConfig.xml file:

  • Agent Admin UI
  • Applications Configuration
  • com.sap.smd.agent.application.wilyhost
  • Application Resources
  • SapAgentConfig.xml
  • Select the affected Diagnostics Agent in the scope >>> DO NOT FORGET <<<
  • Click on Download Custom Resource

If the path is wrong, check the paths provided in step Enter Landscape Parameters in the Managed System Configuration for the affected system and then run the activity Introscope Host Adapter of the Managed System Configuration for the affected system again.

As the SAP GC data is collected from a file, then it is also importand to ensure that the OS permissions are correct. The note 1163751 shows the required OS permissions for Root Cause Analysis.

1163751 - E2E Root Cause Analysis required standard UMASK 027

Also, refer to notes 1833171 and 2127579 and check if the LM-SERVICE patch level described in this note can be applied in the Solman. This patch level deliver a fix for the SAP GC action.

1833171 - Introscope Host Adapter Setup compatibility

2127579 - SAP GC action missing in Introscope after SP update

6.3) Alert for Metric Type "Java Memory Usage" and Metric Name EFFECTIVE OLD SPACE USAGE (%)

In the EWA the following alert is raised:

  • No performance data is returned from BW InfoCube. Infocube: 0SMD_MPEH , metric type: JAVA MEMORY USAGE , metric name: EFFECTIVE OLD SPACE USAGE (%)

Firstly, refer to the section 'Alert for metric type JAVA MEMORY USAGE and metric name: % GC TIME (LAST 5 MINUTES)' of this same WIKI page and check if the SAP GC node is available in the introscope for the managed system server.

If the SAP GC node is correctly available, then refer to note 1856399 and check if the algorithm concurrent mark and sweep (CMS - java parameter -XX:+UseConcMarkSweepGC)  is enalbed in the managed system. If this parameter is being used, then the  EFFECTIVE OLD SPACE USAGE (%) cannot be calculated. If parameter -XX:+UseConcMarkSweepGC is being used the SAP GC node will not contain the metric information, in this case refer to KBA 2156918 to solve the issue.

1856399 - Introscope Host Adapter: supported GC parameters

2156918 - Full Garbage Collection / JVM data is missing in Introscope

6.4) Alert for Metric Type "Java Memory Usage" and Metric Name FULLGC FRACTION (%)

In the EWA the following alert is raised:

  • No performance data is returned from BW InfoCube. Infocube: 0SMD_MPEH , metric type: JAVA MEMORY USAGE , metric name: FULLGC FRACTION (%)

Firstly, refer to the section 'Alert for metric type JAVA MEMORY USAGE and metric name: % GC TIME (LAST 5 MINUTES)' of this same WIKI page and check if the SAP GC node is available in the introscope for the managed system server.

If the SAP GC node is correctly available, then refer to note 1856399 and check if the algorithm concurrent mark and sweep (CMS - java parameter -XX:+UseConcMarkSweepGC)  is enalbed in the managed system. If this parameter is being used, then the  FULLGC FRACTION (%) cannot be calculated. If parameter -XX:+UseConcMarkSweepGC is being used the SAP GC node will not contain the metric information, in this case refer to KBA 2156918 to solve the issue.

1856399 - Introscope Host Adapter: supported GC parameters

2156918 - Full Garbage Collection / JVM data is missing in Introscope

6.5) Alert for Hardware Capacity

The section Hardware Utilization Data of the EWA shows the following message:

  • Hardware Capacity checks could not be run successfully due to missing data

The picture below describes the problem:

 

It might be the case that the metrics is missing for only one specific host, the picture above shows that the 'Performance Data' is OK for the first host, but it is missing in the second host.

To troubleshoot this issue, check in the Workload Analysis for the affected system if the Host performance information is correctly available.

The steps to check it are:

  • Transaction SM_WORKCENTER
  • Root Cause Analysis Workcenter
  • End-to-End Analysis
  • Start the Workload Analysis for the affected system
  • Go to tab 'Host'
  • Check if there is data available for metrics 'CPU & Memory Util.' and 'Monitored Processes'. Ensure that there is data in both granularities as mentioned in the beginning of section 'EarlyWatch Specific Checks' of this WIKI.

The following picture describes the 'CPU & Memory Util.' performance data in the Workload Analysis:


 

Ensure that the Workload Analysis -> Host tab contains the 'CPU & Memory Util.' for all hosts that are listed in the EWA report. In the example above, the system has four (4) hosts: orion-a, orion-b, oriora and orisap.

If you find out that the host information is missing, then refer to section 'Host data is missing, e.g.: CPU, DISKs' of this WIKI and check if the 'SAP Oscol' node is correctly available for the affected host(s).

If the 'SAP Oscol' node is correctly available for all hosts of the managed system, then run again the activity 'Extractor Setup' in the Managed System Configuration for the affected system to ensure that all extractors are correctly scheduled for the system. Finally, check if the extractor of 'Type = Host' is correctly running. The screenshot below describes how to find the extractor:

The section 'Check Managed System Extractors' of this wiki shows how to access the Extractor Framework.


  • No labels