Downtime Announcement: Please note the SAP Community Wiki will be unavailable due to a system upgrade on Thursday, September 24th between 6 and 7 AM CEST
Skip to end of metadata
Go to start of metadata

Agent Authentication and Connectivity - Troubleshooting

To localize the cause of any connectivity issue, it should be identified as a server or an agent issue. The clearest indicator here is whether or not other agents can be authenticated. The following issues are very common and easy to solve. If non of the issues listed below correspond to your issue or if the provided solution did not solve it, proceed with checking the Login Module. If this does neither help follow Checking the Log Files.

Collection of Common Issues

IssueSolution

No Diagnostics Agent cannot connect. The agents do not even appear in the Non-authenticated Agents tab.

Process the following steps in the order specified:

  1. A Diagnostics Agent Cannot Connect
  2. Server Certificate Verification Fails
No Diagnostics Agent cannot connect. The agents appear in the Non-authenticated Agents tab, but clicking on "Trust Agent" does not work.

Process the following steps in the order specified:

  1. Trust Agent Does Not Work
  2. Invalid Configuration on P4S Port
A single Diagnostics Agent cannot connect. The agent does not even appear in the Non-authenticated Agents tab.

Process the following steps in the order specified:

  1. A Diagnostics Agent Cannot Connect
A single Diagnostics Agent cannot connect. The agent appears in the Non-authenticated Agents tab, but clicking on "Trust Agent" does not work.See Trust Agent Does Not Work.
Diagnostics Agents can connect via P4 or P4S, but not via MS or MSHTTPS.See Agent Cannot Connect via the Message Server.
Diagnostics Agents can connect via MS, but not via MSHTTPS. I.e. Diagnostics Agents cannot verify the Message Server certificate.See Message Server Certificate Is not in TrustedCAs.
All connection problems are related to a dedicated Solution Manager instance only.See Agents Cannot Connect to a Dedicated Instance.
In a landscape with a high number of Diagnostics Agents, the number of connected agents is unstable, i.e. agents keep loosing their connection to the Solution Manager, but manage to reconnect every once in a while.See Connectivity Instability in Huge Landscapes.
In the Action Center of the Agent Administration the following message is shown: "The Diagnostic Agent can no longer be updated automatically and will cease to function. Reinstall it according to note 1999000."See The Diagnostic Agent Can no Longer Be Updated Automatically.
On a Solution Manager with multiple instances with Diagnostics Agents connected via P4S the agent shows as disconnected on at least one instanceSee Agents Have Problems to Connect with Multiple Instances via P4S

Log-In Module

All the issues described in the following subsection can be solved as described in section Reconfigure the Log-In Module.

Check the State of Log-In Module and the Root Certificate

Open the Agent Security tab of the Agent Administration and make sure that the log-in module has been configured, and that the root certificate used for agent authentication is generated and valid. 

Check the Configuration of the Log-In Module (ClientCertLoginModule)

  1. Check that the log-in module has been configured correctly. In the Netweaver Administrator go to Configuration →  Security → Authentication and Single Sign-On select the component SAP-J2EE-Engine  → com.sap.engine.services.security.server.jaas.ClientCertLoginModule  
    1. In the upgrade scenario it is important that there are two rules for the ClientCertLoginModule (Rule1 is used for the authentication of 7.10 Diagnostics Agents, whereas Rule2 is necessary for 7.20). 
    2. Check the Flag associated to each log-in module. The ClientCertLoginModule should be first, and marked as SUFFICIENT.  No log-in module should be marked as REQUIRED.

SAP-J2EE-Engine

Check that the component SAP-J2EE-Engine is used by the P4 connection. Open the Netweaver Administrator and go to Configuration →  Security → Authentication and Single Sign-OnCheck that the template used for the Policy Configuration "service.naming" is "SAP-J2EE-Engine".

 

Checking the Log Files

Getting the Connection-Related Log Files

In order to troubleshoot any connection issues, the following logs are relevant. They are required to be attached to any ticket on SV-SMG-DIA-SRV-AGT forwarded to the development support.

Diagnostics Agent Logs
    • Download the Diagnostics Agents log files via the Agent Administration → Agent Log Viewer. Select the relevant agent and click on "Download Logs" and click on "Yes, include support logs".
    • Alternatively (e.g. if the agent cannot be accessed via the Agent Administration due to connection issues), on OS level of the relevant agent, execute ./smdsetup.sh supportlogs in the /usr/sap/<SID>/SMDA<instance nr>/script/ directory and find the generated .zip file at /usr/sap/<SID>/SMDA<instance nr>/SMDAgent/supportLogs_<Date>.zip .
    • More details: https://launchpad.support.sap.com/#/notes/0002455767.
Solution Manager Traces
    • In the NetWeaver Administrator got to Troubleshooting → Logs and Traces → Log Viewer. Click on View → Open View → Developer Traces. Afterwards download the logs via Log Files → Download Log Files. In case the Log Viewer does not work check SAP Note 2506964.
    • Alternatively, take the latest¹ *.trc file from /usr/sap/<SID>/J<instance number>/j2ee/cluster/server<node number>/log on OS level of the Solution Manager server.


Identifying the Issue
The Diagnostics Agent's SMDSystem.log containsSolution

Exception during getInitialContext operation. Cannot establish connection to the remote server. No alive connection. Check state of the server

See No Alive Connection.

[DPCServicePushMetricJob.pushSimpleEvents] Error occurred when calling the DPC Push web service. (Endpoint: http://solutionManagerHostName:8000/sap/bc/srt/scs/sap/e2e_dpc_push?sap-client=100).
...
Caused by: java.net.SocketTimeoutException: connect timed out (local port 12345 to address 123.45.67.89(agentsHostName.domain.name), remote port 8000 to address 10.11.12.13(solutionManagerHostName.domain.name))

See DPC Push Fails.
The Solution Manager's developer trace contains 

ConnectionImpl.setConnectionInfo(ConnectionInfo connInfo) : Cannot get information about connection 09 01 00 00 85 ED 3B 00 Error: not known error code received -10

See No Alive Connection.

com.sap.smd.api.util.SMDAgentCertificateGenerator$NoCAonSslPortException: Unable to read CAs

See Unable to Read CAs.

Agent Authentication and Connectivity - Common Issues

This section describes common issues regarding the agent management and their solutions.

A Diagnostics Agent Cannot Connect

If the Diagnostics Agent is not visible at all in the Agent Administration (neither in the "Connected Agents" tab nor in the "Non-Authenticated Agents" tab), a reconfiguration of the connection parameters might be necessary:

In the file system of the Diagnostics Agent's host go to /usr/sap/<SID>/SMDA<instance nr>/script/ and execute the following command: 

 ./smdsetup managingconf hostname:"sapms://host" port:"port"

The host and port parameters can be found at Solution Manager Workcenter → Infrastructure → Framework → Agent Framework:


Wait a few minutes and check the "Connected Agents" tab nor in the "Non-Authenticated Agents" tab for the agent. If applicable, click on "Trust Agent". If the agent does still not occur in the Agent Administration, ensure that the connection URLs specified in the runtime.properties of the Diagnostics Agent match the pattern described at the section Connection Types.

Trust Agent Does not Work

If not already done, configure the Solution Manager to support the authentication of Diagnostics Agents via certificate as follows:

  1. Make sure that each Solution Manager AS Java instance must not have more than one node. Multi-node instances cause serious agent connectivity issues, e.g. the "Trust Agent" functionality will not work.
  2. If the version of the agent (shown in the "Non-authenticated Agents" tab) and the LM-SERVICE version differ, select the agent and click on Update Agent. Once the agent has been up- or downgraded, click on "Trust Agent(s)".

  3. If this does not help, go to the installation directory of the affected agent node and delete configuration/secstore.properties and configuration/master_password (if existing), set smdserver.connection.requiresAuthentication=verified-certificate in the configuration/runtime.properties and restart the agent to reset all configuration and client certificates and get it back into the non-authenticated mode. Afterwards, authenticate the agent by clicking on "Trust Agent" in the Agent Administration.

  4. Set the TrustedP4S property within NWA. For more information, see SAP Note 2013578.
  5. In the SOLMAN_SETUP transaction, under Infrastructure Preparation, go to step 2.3 ("Diagnostics Agent Authentication"). Click on "Generate new certificate", and "Save".
  6. Restart the AS Java.
  7. Again try to Trust the agent.
  8. If the agent does still not appear in the "Connected Agents" tab, enable the Connection Log. If the Connection Log contains the error message shown below, check that the root certificate for agent authentication has been generated on the server, and note this certificate’s properties. 

  9. Navigate to Configuration → Security → Certificates and Keys and check the key stores TrustedCAs as well as SMDAgentSecurity. If the P4S Status displayed in the Agent Security tab of the Agent Administration is red (error message "The CAs associated with the P4S ports do not match"), go to the server’s NetWeaver Administrator and navigate to Configuration → Security → Certificates and Keys. Verify that the first SMD-CA* certificate of the SMDAgentSecurity key store is valid and equal to one of the SMD-CA* certificate(s) at TrustedCAs (e.g. by comparing the checksums or validity dates). If not, delete or rather re-generate the certificates accordingly. Afterwards, again trust the agent. Whilst SMDAgentSecurity is used during the certificate generation, TrustedCAs is used during the verification.

  10. If the agent does still not appear in the "Connected Agents" tab, navigate to /usr/sap/<SID>/SMDA<Instance-ID>/script on the agent's machine and issue the command 
    smdsetup certificate operation:"LIST_CA_ALIAS"
  11. For each alias name, issue the command
    smdsetup certificate operation:"PRINT"  alias:"p4s_client_cert_<number>"
    and compare the subject name of the certificate to the issuer of the server’s SSL certificates, to be obtained as described in step 5 and check for inconsistencies.
  12. If the agent can still not connect, open the Action Center and create a screen shot of the displayed issues. Furthermore, create screen shots of the TrustedCAs and the SMDAgentSecurity entries in the "Certificates and Keys" view of the NetWeaver Administrator. Enable server-side debug logging and again click on trust agent. Download the Solution Manager AS Java server logs. Provide all the screen shots and the server logs to the SAP support.
  13. If the agent cannot connect after trusting it due to "Exception during getInitialContext operation. Wrong security principal/credentials", check the authentication stack for "service.naming". For this stack, you don't need the logon ticket modules because they are for the HTTP communication while the "service.naming" is for the Java RMI/P4 communication. Therefore, change the "service.naming" stack as follow:

    1. Logon to NetWeaver Administrator and navigate to "Authentication and Single-Sign On".
    2. Find the stack service.naming, you may use the filter Type = Service at the right side.
    3. Go to Edit mode and remove the Used Template → select the blank.
    4. Now, you may remove the unnecessary login modules and leave only the following:
      1. ClientCertLoginModule      SUFFICIENT  (keep the same options)
      2. BasicPasswordLoginModule    REQUISITE
    5. Save the changes and check if it is working. 

Credentials not Found

The following exception in the Diagnostics Agent log indicates that the Diagnostics Agent is still configured to use basic authentication (smdserver.connection.requiresAuthentication in the runtime.properties is case-insensitively set to "true", "yes", or "basic" or it is not set at all), which is not supported anymore. The agent should appear in the "non-authenticated agents" tab. Clicking on "Trust Agent" should make the agent switch to certificate-based authentication (smdserver.connection.requiresAuthentication=verified-certificate). If you still see the exception in the logs of the Diagnostics Agent, the "Trust Agent" operation has not been executed successfully. Figure out why the "Trust Agent" has not been executed successfully by following Trust Agent Does not Work.

 

javax.naming.NoPermissionException: Credentials not found, the configuration is empty.
at com.sap.smd.agent.connection.SMDConnector.checkAuthenticationSettings(SMDConnector.java:1129)
at com.sap.smd.agent.connection.SMDConnectionTask.internalAttemptConnection(SMDConnectionTask.java:537)
at com.sap.smd.agent.connection.SMDConnectionTask$1.run(SMDConnectionTask.java:106)
at com.sap.smd.agent.connection.P4JNDIContextHelper.executeInSecurityContext(P4JNDIContextHelper.java:150)
at com.sap.smd.agent.connection.SMDConnectionTask.attemptConnection(SMDConnectionTask.java:101)
at com.sap.smd.agent.connection.SMDConnectionTask.run(SMDConnectionTask.java:1321)
at java.lang.Thread.run(Thread.java:763)

Server Certificate Issues

Server Certificate Verification Fails

If the agent is not only not connect, but does also not appear in the Non-Authenticated Agents tab, there might be an issue with the server certificate of the P4S port. This can be checked by applying SAP Note 2528155 and setting smd.agent.p4client.checkServerCertificate=false in the agent's runtime.properties and restarting the agent. If the agent can now connect, it was having trouble verifying the server.

Correcting Problems at the P4S Port or rather the Server Certificate 

  1. Modify the server's P4S port to use a certificate that is trusted by the agents. If you have recently modified the certificate, it may be possible to use the previous certificate. The standard certificates delivered with the agents' JVM will also be trusted (including SAPNet, etc.). 
  2. Once the agents can connect again, do the following:
    1. Use the Agent Administration UI to enter the Maintenance Mode.
    2. Go to the Agent Security tab and switch off the verification of the server certificate. 
    3. Switching off the Maintenance Mode will then allow the agents to reconnect and the new property disabling the verification will be communicated to the agents.
    4. Switch on the Maintenance Mode again.
    5. Deploy the new certificate at the P4S port. 
    6. Then switch off the Maintenance Mode, and the agents will reconnect. 

 Invalid Configuration on P4S Port

If the following symptoms apply, there might be an issue with the server certificate associated to the P4S port:

  1. The Diagnostics Agents appear in the "Non-authenticated Agents" tab of the Agent Administration, but clicking on "Trust Agent" does not work for any agent.
  2. The error message "Invalid configuration on P4S port" in the Action Center of the Agent Administration.
  3. The "P4S Status" shown in the "Agent Security" tab is "The CAs associated with the P4S ports do not match".

To (temporarily) solve this issue, apply SAP Note 2528155. You can also use this note to verify issue's cause. As a long-term solution, see 2816888 or fix the SSL setup of your Solution Manager AS Java as described at Preparing the AS Java.

Self-Signed Certificate on the P4S Port

If the Diagnostics Agent cannot connect to the Solution Manager because the "Trust Agent" action failed with the following exception in the Solution Manager AS Java's default traces

SSL port examiner action failed.
[EXCEPTION] com.sap.smd.api.util.SMDAgentCertificateGenerator$SelfSignedCertificateException: ICM_SSL_123456_7890

go to the NetWeaver Administrator → Configuration → Security → SSL and select the SSL Access Point "P4SEC". Notice the "Details" section of the private key entry "ssl-credentials".

Solution

Replace the above mentioned certificate by a certificate that is not self-signed, i.e. ensure that more than one certificate is displayed in the "Details" section.

Incomplete Certificate Chain

Symptom
  • The "P4S Status" shown in the "Agent Security" tab reads 
    • "The certificate chain associated with the key ssl-credentials in the keystore ICM_SSL_*****_*****" is incomplete"
    • or "The certificate chain associated with the key ssl-credentials in the keystore {0} is incomplete"
  • The traces of the Solution Manager AS Java server contain the following error message:

 

[ServerCAValidityCheck] incomplete chain on P4S port
[EXCEPTION]
com.sap.smd.api.util.SMDAgentCertificateGenerator$IncompleteChainException: ICM_SSL_*****_*****
at com.sap.smd.api.util.SMDAgentCertificateGenerator.getActiveChainFromKeyStore(SMDAgentCertificateGenerator.java:554)
at com.sap.smd.api.util.SMDAgentCertificateGenerator.getCAsFromKeyStore(SMDAgentCertificateGenerator.java:578)
at com.sap.smd.api.util.SMDAgentCertificateGenerator.access$300(SMDAgentCertificateGenerator.java:58)
at com.sap.smd.api.util.SMDAgentCertificateGenerator$SslPortExaminerAction.run(SMDAgentCertificateGenerator.java:445)
at com.sap.smd.api.util.SMDAgentCertificateGenerator$SslPortExaminerAction.run(SMDAgentCertificateGenerator.java:434)
at java.security.AccessController.doPrivileged(Native Method)
at com.sap.smd.api.util.SMDAgentCertificateGenerator.getCertificateIssuers(SMDAgentCertificateGenerator.java:487)
at com.sap.smd.api.util.SMDAgentCertificateGenerator.getActiveSslCertificate(SMDAgentCertificateGenerator.java:428)
at com.sap.smd.server.health.check.ServerCAValidityCheck.doCheck(ServerCAValidityCheck.java:25)
at com.sap.smd.server.health.ServerHealthRunner.run(ServerHealthRunner.java:66)
at com.sap.smd.server.exec.TaskRunner.run(TaskRunner.java:47)
at com.sap.smd.server.util.concurrent.PooledExecutor$Worker.run(PooledExecutor.java:785)
at java.lang.Thread.run(Thread.java:763)
Recommended Solution 

Apply SAP Note 2693854

Alternative Solution
  1. In the NetWeaver Administrator of the Solution Manager AS Java, navigate to Configuration → Security → SSL  and notice the name of the key store view associated to the P4SEC port of the AS Java instance in question. Usually this name starts with "ICM_SSL_".
  2. Navigate to Configuration → Security → Certificates and Keys and selected the mentioned key store view.
  3. Recognize the first entry of this key store view (usually the alias name starts with "SMD-CA_"). This certificate chain is used by the connect Diagnostics Agents to generate their client certificates and to verify the Solution Manager AS Java's identity. Mind the following misconfigurations that might cause the issue:
    1. If there are entries before the first "SMD-CA_" entry or if there is no "SMD-CA_*" entry at all, this is very likely not what you want. Back-up the key store view using the export functionality and remove all entries except the "SMD-CA_" entry you want to be used for the P4S port.
    2. If the alias name of the first entry starts with "SMD-CA_", i.e. the first entry most certainly is the certificate chain you want to be used for the agent management, ensure that the chain of this entry is complete by checking the details section of this entry. Ensure that the last item is a root certificate, that is its subject DN equals its issue DN. Fix the incomplete chain by importing the missing certificate(s) or generate a new chain.

Set Up P4S

If there is no P4S port configured on the Solution Manager AS Java some configuration regarding the certificate generation and authentication might be incomplete. If you are facing any issue connecting an Diagnostics Agent to the Solution Manager, setup SSL and P4S on the Solution Manager AS Java by following 17705852419031, and 2268643. Once the P4S port is available, reconfigure the Diagnostics Agent to connect to the P4S port by adjusting the runtime.properties as described at Connection Types.

Agents Have Problems to Connect with Multiple Instances via P4S

Symptoms

You are running a Solution Manager with multiple instances connecting the Diagnostics Agent via P4S the agent shows as disconnected on at least one instance and as connect on at least one instance. In the Developer Traces of the Solution Manager error messages relating to the P4 connection of the agent should be present.

Reason

The agent only allows one P4S connection of one Solution Manager in some configurations.

Recommended Solution

Upgrade your LM-SERVICE to SP 7 patch 8, SP 8 patch 3 or higher (see SAP Note 2746956).

Work-a-round

Alternatively, go to the Agent Administration and enable the Maintenance Mode. Next, go to Advanced Settings. Look for the parameter single.agent.connection and set it to false. Save and Apply these settings. Go back to the first Tab and disable the Maintenance Mode. Due to the connection errors there seem to be problems to promote the setting change to the agents. If the problems persist for an agent, please restart the agent. If the setting still cannot be promoted to the agent, go to SMDAgent/configuration/runtime.properties and replace "single.agent.connect=true" with "single.agent.connect=false".

Agents Cannot Connect via the Message Server

If a Diagnostics Agent cannot connect to its Solution Manager AS Java via a Message Server, check whether it can connect via a direct P4 or rather P4S connection by adjusting the connection URL in the runtime.properties as follows. Replace

smd.agent.connection.url=ms\://your-solution-manager-java.home\:8101/P4

by

smd.agent.connection.url=p4\://your-solution-manager-java.home\:50004

(adjust the host and domain names as well as the port numbers accordingly) and restart the agent. If the agent can connect via P4 or P4S protocol, but not via MS or MSHTTPS, this indicates a problem with the Message Server. Make sure that the Message Server is correctly configured. If the agent is using MS/HTTPS, examine the output of the URL https:// java-server-host :44402/msgserver/text/logon (adjust the port number as appropriate, 8101 or 8102 for MS/HTTP). Examine the output of this link to assure that the P4 and P4S ports are correct.

Connection via the Message Server Fails due to an UnknownHostException

If the connection via the Message Server fails due to an UnknownHostException similar to the following, adjust the Message Server configuration such that it provides the full qualified host name for the P4(S) port.

URL=ms://myhost.my.domain:8102/P4, protocol=P4(None), Reason: java.net.UnknownHostException: myhost;

Open http://myhost.my.domain:8102/msgserver/text/logon in a browser and check the host name of the P4(S) port. If only the short host name is provided, the Diagnostics Agent might not be able to connect to the P4(S) port. The expected output is as follows:

 

...
P4	myhost.my.domain	50404	LB=1
P4S	myhost.my.domain	50405	LB=1
...

Message Server Certificate Is not in TrustedCAs

If the agent can connect via MS, but not via MSHTTPS and the SMDSystem.*.log contains the error message "Cannot check server certificate on connect: Message server certificate not in TrustedCAs", there is an issue with the server certificate of the Message Server.

Solution

The Diagnostics Agent is only using certificate chains of the key store view associated with the P4SEC port and self-signed certificates of the TrustedCAs key store view to verify the Message Server or rather the ICM. Check the SMDSystem.*.log to identify the certificate the Message Server is using. The Message Server certificate can be a self-signed of the TrustedCAs key store view. Go to Configuration → Security → Certificates and Keys and verify that TrustedCAs key store view contains the message server certificate stated in the SMDSystem.*.log. 
Alternatively, the Message Server can use the signed certificate (not self-signed) of the ICM P4S port. In the NetWeaver Administrator, go to Configuration → Security → SSL and ensure that the identified certificate and its complete issuer chain is part of the key store view associated to the P4SEC port of the regarding SAP instance. I.e., ensure that the Message Server is using the same certificate as the ICM P4S port, which must not be self-signed.

Agents Cannot Connect to a Dedicated Instance

On a cluster system, it is important to test the connectivity using a direct P4 or P4S connection to each instance and thereby determine if the problem is only related to one instance. Furthermore ensure that the agents are visible in the Agent Adminstration application on all instances. If the connectivity problem is related to a specific instance, verify in NetWeaver Administrator  Start and Stop Operations that the AS Java Service "smd~service" is started on every instance.

Connectivity Instability in Huge Landscapes 

Each Diagnostics Agent increases the load within the SAP J2EE Engine used as part of the Solution Manager technology stack. When connecting several hundreds of Diagnostics Agents, the below described adjustments are required. Furthermore, it is highly recommended to upgrade to the latest patch level of SP 7 or any higher SP as this version includes some fundamental improvements regarding the connectivity management of Diagnostics Agents. You need also to make sure that each Solution Manager AS Java instance must not have more than one node. Multi-node instances cause serious synchronization issues regarding the connected Diagnostics Agents between the nodes. However, customers who connect more than 1,000 agents often reach a limit of the SAP J2EE Engine used in SAP Solution Manager. As each customer situation is different, SAP cannot exactly specify how many connected agents an individual Solution Manager system can take. SAP in general recommends to either lower the number of Diagnostics Agents (e.g. by dividing the agents to multiple Solution Managers) or to use Focused Run for SAP Solution Manager, which was dedicatedly developed to satisfy advanced customer needs in terms of performance, scalability, security, and automation.

  1. Use the Quick Sizer Tool to calculate CPU, disk, memory and I/O resources for the Solution Manager. 
  2. Access the Agent Administration user interface at https://j2ee-host:port/smd/AgentAdmin.
  3. Switch on the Maintenance Mode.
  4. Change the parameters according to the attached Excel document.

  5. Restart the Solution Manager Java by executing the commands  stopsap  and  startsap ALL  using the user  <SID>adm. To restart individual Java instances you can use the command line tool jcmon pf=<instance_profile_file>  and within this tool execute the commands 20 and 19 and finally repeat executing the command 1 until all JEE processes are running again.

  6. Turn off the Maintenance Mode. 

During an Update

As the update of the LM-SERVICE component in huge landscapes generates an extra load on the AS Java, it is recommended to restricted the number of connected Diagnostics Agents during an update by introducing the agents tier-wise in batches of 30-50 managed systems (by shutting down the other agents). After the Solution Manager server has updated all agents, the restriction can be dissolved.

Recommended SAP Notes

As reasoned above, it is highly recommended to upgrade to SP 7 in order to stabilise the agent connectivity. If this is not possible, apply the following SAP Notes to reduce the load on the AS Java:

  • SAP Note 2599110 (SP 6) provides lightweight pings from the Diagnostics Agent to the Solution Manager, which reduces the load on the AS Java.
  • SAP Note 2606713 (SP 5 and 6) fixes ping strategy functionality. Until now: A Diagnostics Agent will be disconnected right after a single ping failure. Most likely this would happen if there is already an ongoing issue and therefore putting even more load on the system. The fix will allow tolerating a dedicated number of ping failures before the reconnection procedure is launched.

Analyzing the P4 Thread Usage

In order to analyze the P4 thread usage log in to the Solution Manager Java system and use the "P4 Information Command":

telnet localhost 5<instance number>08
add p4
p4info

If the displayed value of the "Thread usage" stays at 100% for several minutes the connection-related parameters need to be adjusted further.

Multiple IP Addresses for One Solution Manager P4/P4S Port

Running an AS Java with P4/P4S ports associated to more than one IP address many cause issues and introduces instability in the RMI protocol. If you do not have a special purpose, do not maintain multiple IP addresses for the P4 and P4S ports.

Problems are usually indicated by an error message in the Developer Traces in the NWA.

The JNDI lookup or rather the P4 narrow of mediated/*****@1999.01.01-11.11.11.111 did not return within 2000ms


If there are threads hanging when requesting outbound connections from the ICM of the Solution Manager AS Java to a Diagnostics Agent, it is very likely caused by (accidentally) misconfigured P4/P4S IP addresses. Ensure that the P4 or rather the P4S port of each instance is associated to one IP address only. Make sure the IP address you choose is visible by all Diagnostics Agents. Initially, your default profile might read as follows:

...
icm/server_port_0 = PORT=5$$00,PROT=HTTP,TIMEOUT=7200,PROCTIMEOUT=1800
icm/server_port_1 = PORT=5$$04,PROT=P4,TIMEOUT=1800,PROCTIMEOUT=1800
icm/server_port_2 = PORT=5$$06,PROT=P4SEC,TIMEOUT=1800,PROCTIMEOUT=1800
...

To only use one IP address per instance, add the following lines to the instance profile of the affected Solution Manager AS Java instance(s), replacing "123.45.67.89" by the desired IP address. Keep the default profile as it is.

icm/server_port_0 = PROT=HTTP,PORT=5$(SAPSYSTEM)00,TIMEOUT=7200,PROCTIMEOUT=1800
icm/server_port_1 = PROT=P4,PORT=5$(SAPSYSTEM)04,HOST=123.45.67.89,TIMEOUT=1800,PROCTIMEOUT=1800
icm/server_port_2 = PROT=P4SEC,PORT=5$(SAPSYSTEM)06,VCLIENT=0,HOST=123.45.67.89,TIMEOUT=1800,PROCTIMEOUT=1800
icm/server_port_3 = PROT=IIOP,PORT=5$(SAPSYSTEM)07
icm/server_port_4 = PROT=TELNET,PORT=5$(SAPSYSTEM)08,HOST=localhost
icm/server_port_5 = PROT=P4,PORT=5$(SAPSYSTEM)04,HOST=127.0.0.1


Log-In Module Issues

If the log-in module has not been configured correctly, it is very likely that the step 2 .3  "Diagnostics Agent Authentication" of the "Infrastructure Preparation" has not been executed successfully. Open the transaction "solman_setup" , go to "Infrastructure Preparation" → "2 . Set Up Java Connectivity" → "2 .3 Diagnostics Agent Authentication and resolve the issues stated in the logs.

As a work-around the Web Service that configures the log-in module can be triggered manually (This is not officially supported.):

  1. Open the Web Services Navigator (https://j2ee-host:port/wsnavigator)
  2.  Search for the provider system SMDSetupAuthenticationViImpl.
  3. Select updateLoginModule.
  4. Provide the credentials and execute the Web Service call.

No Alive Connection

If the Diagnostics Agent's SMDSystem.log contains the following error message, 

[EXCEPTION]
javax.naming.NamingException: Exception during getInitialContext operation. Cannot establish connection to the remote server. [Root exception is com.sap.engine.services.rmi_p4.P4IOException: No alive connection. Check state of the server] [...]

The Solution Manager's developer trace contains plenty of errors like:

ConnectionImpl.setConnectionInfo(ConnectionInfo connInfo) : Cannot get information about connection 09 01 00 00 85 ED 3B 00 Error: not known error code received -10

The solution is to restart the Solution Manager Java server.

Unable to Read CAs

The SAP Solution Manager 7.2 default trace shows the following error every 10 minutes:

#2.0##2017 01 31 16:24:06:701#0-200#Error#com.sap.smd.server#
#SV-SMG-DIA-SRV-AGT#tc~smd~server~service#............Thread[ExRun:SMD_MGR_49,5,SMD_MGR:ExecTG]#Plain##
[ServerCAValidityCheck]
[EXCEPTION]
com.sap.smd.api.util.SMDAgentCertificateGenerator$NoCAonSslPortException: Unable to read CAs [...]

This issue can be solved as described in SAP Note 2423083.

Diagnostic Agent Can no Longer Be Updated Automatically

If the error message stated below is shown in the Agent Administration or rather the Action Center a reinstallation of the affected Diagnostics Agents is required. Please proceed as described in SAP Note 1999000

The Diagnostic Agent can no longer be updated automatically and will cease to function. Reinstall it according to note 1999000.

False Positive

In very rare cases the error message might persist although the Diagnostics Agent has been updated to version 7.20 without any issues. This case can be distinguished by checking the version of the affected Diagnostics Agent using the "Detailed" view in the Agent Administration. If and only if the error message is shown for an agent having a version ≥ 7.20, the error message is irrelevant and can either be ignored without any consequences or eliminate by proceeding as follows. The root cause for this behaviour is old zombie entries of agents whose server name has changed anytime before. The following procedures describes how to manually delete this odd entries. "Automatically Remove the Zombie Entries" is recommended.

Automatically Remove the Zombie Entries
  1. Open the Agent Administration (transaction sm_workcenter → SAP Solution Manager Administration → Agents Administration → Agent Admin (All Agents))
  2. Click on "Show Offline Agents" in the "Connected Agents" tab.
  3. Click on "Remove Offline Agent Entries".
  4. Wait a few minutes.
  5. Go to SAP Solution Manager Administration → Overview → Landscape → Agent Framework → Status and click on "Refresh" 
  6. The status of the affected agents should have switched from INCOMPATIBLE VERSION to will change to STARTED. 
Manually Remove the Zombie Entries

If you cannot use the "Remove Offline Agent Entries" functionality (e.g. because there are currently disconnected agents you want to keep), you can proceed as follows;

  1. Identify the affected Diagnostics Agent:
    1. In the Agent Administration open the Action Center and remember the Agent ID shown in the "scope" column. E.g.:

      host00_DAA_SMDA98
  2. Start the Config Tool
    1. On Windows, execute

      C:\usr\sap\<SID>\J<Instance>\j2ee\configtool\configtool.bat
    2. On Linux, execute the following. If you are using a remote connection (SSH) remember to enable X11 forwarding.

      cd /usr/sap/<SID>/J<Instance>/j2ee/configtool
      ./configtool.sh
  3. Download the knownAgents file:

    1. In the ConfigTool click Tools → Configuration Editor.

    2. Expand sm_diagnotics_data → smdserver → com.sap.smd.server

    3. Double click knownAgents and download it.

  4. Remove the zombie entries of dedicated agent:
    1. Backup of the knownAgents file and keep.
    2. In the knownAgents file, search for the AgentID of the dedicated agent. 
    3. If there are two or more keys in the properties file having the same Agent ID the root cause described above applies. Please proceed.
      If not, this procedure does not apply to your case and you can stop here. Please create an incident using the component SV-SMG-DIA-SRV-AGT.
    4. Identify and remember all (most likely two) key-prefixs (the part of the key before "~"). In the example below, the prefixes are host00 and host01.

      host00~agentId=host00_DAA_SMDA98
      ...
      host01~agentId=host00_DAA_SMDA98
    5. Search for both key-prefixes and identify the zombi agent entry by looking for the old version (7.10). Here the zombi entry is host00.

      host00~version=7.10.12.5.20150811111924
      ...
      host01~version=7.20.3.0.20160811041533
    6. Delete all lines starting with the key-prefix related to the zombi entry. E.g.:

      host00~ipAddress=123.4.5.6
      host00~lastConnection=Fri Apr 01 12\:42\:00 EDT 2016
      host00~canonicalName=host00
      host00~agentId=host00_DAA_SMDA98
      host00~localName=host00
      host00~serverName=host00
      host00~shortName=host00
      host00~version=7.10.12.5.20150811111924
    7. Save the file.

  5. Upload it to the Solution Manager using the ConfigTool:

    1. Use the small button (tooltip "Switch between view and edit mode") to enable the upload functionality. 

    2. Upload the edited files.

    3. Restart the Java server.

  6. The error message should not be displayed anymore. In case of any issue, you can revert everything by uploading the backup of the original knownAgents file.

Agent Cannot Connect Via SAP Router

The following error in the SMDSystem.*.log indicates that there is an issue with the connection between the Diagnostics Agent and the Solution Manager via an SAP Router.

Connecting to SMD server ms://14.15.16.17:8102/P4 failed - error counter: 1
[EXCEPTION]
javax.naming.NamingException: Exception while trying to get InitialContext. [Root exception is java.io.IOException: Cannot get Socket. Reason:Cannot create NI socket with router string: /H/10.11.12.13/S/3200/H/14.15.16.17/S/8102/P/null]
	[...]
Caused by: java.io.IOException: Cannot get Socket. Reason:Cannot create NI socket with router string: /H/10.11.12.13/S/3200/H/14.15.16.17/S/8102/P/null
	[...]

Assuming that 10.11.12.13 is the IP address and 3200 the port number of the SAP Router and that the Solution Manager is to be reached at 14.15.16.17:57000, search the log for an entry similar to

SAP Router  configured for P4 connection with route /H/10.11.12.13/S/3200.

or rather

SAP Router with password configured for P4 connection with route /H/10.11.12.13/S/3200/H/14.15.16.17/S/57000.

Ensure that the stated SAP Router settings are correct and use the "smdsetup addsaprouter route" and "smdsetup addsaprouter pass" functionallity of the Diagnostics Agent to adjust the SAP Router settings accordingly. The smdsetup script can be found at /usr/sap/DAA/SMDA98/script. For more information see Using the SMD Setup Script and section "7.13 SAP Router" of  Installation of Diagnostics Agent on UNIX and Linux. Furthermore, ensure that the Solution Manager's P4 port (e.g. 50004) is in the SAP Router table.

Afterwards, verify the settings by checking the properties "smd.agent.connection.saprouter" and "smd.agent.connection.transport" at usr/sap/DAA/SMDA98/SMDAgent/configuration/runtime.properties.

Agent Cannot Connect Via P4S Through SAP Router

Solution

Update your LM-SERVICE to a version higher than SP 3 Patch 2, SP 4 Patch 2 or rather SP 5 Patch 0. Also see SAP Note 2458281.

Issues with the Introscope Host Adapter and Byte Code Agent via SAP Router

See https://launchpad.support.sap.com/#/notes/1874044.

Cannot Handle sun.security.x509.X509CertImpl Certificates

While trusting a Diagnostics Agent in the Agent Administration UI, the following error is reported: "java.rmi.RemoteException: Cannot handle certs of class class sun.security.x509.X509CertImpl".

Furthermore, the following entries can be seen in logs of the Diagnostics Agent:

java.rmi.RemoteException: Could not generate certificate; nested exception is: 
java.rmi.RemoteException: Cannot handle certs of class class sun.security.x509.X509CertImpl
at com.sap.smd.agent.connection.configuration.ManagingConnectionManipulator.generateCertificate(ManagingConnectionManipulator.java:1365)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)

java.lang.IllegalArgumentException: argument type mismatch
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)

or

java.lang.IllegalArgumentException: argument type mismatch
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)

To solve this issue, apply SAP Note 2719280. 

Peer certificate rejected by ChainVerifier

The Diagnostics Agent cannot connect to the Solution Manager AS Java due to the following error:

Feb 13, 2019 10:19:06 AM [Thread[Connector,5,main]                     ] Error      Connecting to SMD server mshttps://my.sol.man.sap.com:50002/P4S failed - error counter: 1
[EXCEPTION]
javax.naming.NamingException: Exception while trying to get InitialContext. [Root exception is iaik.security.ssl.SSLCertificateException: Peer certificate rejected by ChainVerifier]
	at com.sap.engine.services.jndi.InitialContextFactoryImpl.getInitialContext(InitialContextFactoryImpl.java:386)
	at javax.naming.spi.NamingManager.getInitialContext(NamingManager.java:684)
	at javax.naming.InitialContext.getDefaultInitCtx(InitialContext.java:313)
	at javax.naming.InitialContext.init(InitialContext.java:244)
	at javax.naming.InitialContext.<init>(InitialContext.java:216)
	at com.sap.smd.agent.connection.SMDConnectionTask.getNewSMDInitialContext(SMDConnectionTask.java:194)
	at com.sap.smd.agent.connection.SMDConnectionTask.getTransientInitialContext(SMDConnectionTask.java:649)
	at com.sap.smd.agent.connection.SMDConnectionTask.internalAttemptConnection(SMDConnectionTask.java:302)
	at com.sap.smd.agent.connection.SMDConnectionTask.access$000(SMDConnectionTask.java:54)
	at com.sap.smd.agent.connection.SMDConnectionTask$1.run(SMDConnectionTask.java:96)
	at com.sap.smd.agent.connection.P4JNDIContextHelper.executeInSecurityContext(P4JNDIContextHelper.java:141)
	at com.sap.smd.agent.connection.SMDConnectionTask.attemptConnection(SMDConnectionTask.java:91)
	at com.sap.smd.agent.connection.SMDConnectionTask.run(SMDConnectionTask.java:1062)
	at java.lang.Thread.run(Thread.java:836)
Caused by: iaik.security.ssl.SSLCertificateException: Peer certificate rejected by ChainVerifier
	at iaik.security.ssl.y.a(SourceFile:932)
	at iaik.security.ssl.n.b(SourceFile:1066)
	at iaik.security.ssl.n.a(SourceFile:1503)
	at iaik.security.ssl.y.d(SourceFile:784)
	at iaik.security.ssl.SSLTransport.startHandshake(SourceFile:569)
	at iaik.security.ssl.SSLTransport.getOutputStream(SourceFile:648)
	at iaik.security.ssl.SSLSocket.getOutputStream(SourceFile:391)
	at com.sap.engine.interfaces.cross.LoadBalancerImpl.staticGetAllAccessPoints(LoadBalancerImpl.java:142)
	at com.sap.engine.interfaces.cross.CrossObjectBroker.getDestination(CrossObjectBroker.java:168)
	at com.sap.engine.services.jndi.InitialContextFactoryImpl.getInitialContext(InitialContextFactoryImpl.java:293)
	... 13 more

Solution

The Diagnostics Agent might not be able to initially connect to the Solution Manager AS Java via P4S due to missing certificates. Temporarily connect it via P4 using the smdsetup script as described at Trust Agent Does Not Work and click on "Trust Agent". Afterwards switch the connection to the intended type using the Agent Administration's Agent Connectivity tab.

Update List of Trusted CAs

If the Diagnostics Agent are configured to verify the Solution Manager's server certificate (Agent Administration → Agent Security → Server Authentication), it might be necessary to update the agent's trust store (i.e. if the Solution Manager server's certificate changed). This can be done by using  "Update list of trusted CAs" button at the Agent Security tab of the Agent Administration. The CA of the SSL certificate associated with the P4S port and all certificates in the TrustedCA keystore view will be sent to all connected agents. Mind that the button is only available if verification of the SSL certificate is enabled as there is no need to update the trust stores if the agents do not verify the server's certificate.

 

 

Agent Authentication and Connectivity - Diagnostics Agent to Managed System Communication Issues

Diagnostics Agent Cannot Establish a P4 Connection to the Managed System

If the SMDAgentApplication.*.log contains one of the following error messages, there might be an issue with the authentication of the Diagnostics Agent to the managed system's P4 port:

Error      P4 connection error to SAP system [SID/00]
[EXCEPTION]
javax.naming.NoPermissionException: Exception during getInitialContext operation. Wrong security principal/credentials. [Root exception is com.sap.engine.services.security.exceptions.BaseLoginException: Cannot authenticate the user.] [...]
Error      License Management - Unexpected error during service initialization
[EXCEPTION]
com.sap.smdagent.plugins.connectors.p4.exceptions.P4AuthorizationException: Access is denied to SAP System sid [SID/00]: check the connection credentials.More details about the error in agent 'xyz' log file (SMDAgentApplication.X.log).; nested exception is: 
	com.sap.engine.services.security.exceptions.BaseLoginException: Cannot authenticate the user. [...]
License Management - Failed to distribute Licenses to System: SID - The targeted system is stopped [...]

Solution

The user SM_COLL_SID that is used to access the managed system and collect data is missing some rights. Ensure that user SM_COLL_SID it is not locked. Verify that the user SM_COLL_SID is added to the "Administrators" group in NetWeaver Administrator → Security Provider →  Policy Configurations (select "SAP-J2EE-Engine") → Security Roles → select "Administrators" (details at User Management of the Application Server Java). If the managed system is a double stack, remove SM_COLL_SID from user list and ensure that it is a member of the SAP_J2EE_ADMIN group. If the user SM_COLL_SID is marked as "created manually" at step 6 "Create Users" of the Managed System Configuration, any issue with the user will not be flagged, because you certified that the user is valid and has no issues. 

Further details at Managed System Checklist and Troubleshooting Diagnostics Configuration for Java Systems → Step 2.

Agent Authentication and Connectivity - Java (Solution Manager or Diagnostics Agent) to ABAP Communication Issues

DCC Push Issues

DCC Push is used by the Solution Manager AS Java to send the current connection status of a Diagnostics Agent to the Solution Manager AS ABAP in order to trigger dedicated alerts if applicable. If there is one of the following exceptions in the server traces of the Solution Manager AS Java, the DCC Push functionality is not working. 

[DCCAgentStatusPush.register] Error to push agent event myAgentsHostName
[EXCEPTION]
This exception is wrapper of javax.xml.ws.soap.SOAPFaultException. com.sap.engine.services.webservices.espbase.client.bindings.exceptions.SOAPFaultException: Authorization missing for service "urn:sap-com:document:sap:soap:functions:mc-style E2E_DCC_PUSH", operation "E2eAfwkPushStatus"; more details in the web service error log on provider side (UTC timestamp 20181008134026; Transaction ID 37f693a6da461006bd0fe4e294d08d71)
at com.sap.engine.services.webservices.espbase.client.bindings.impl.JAXWSUtil.processFault(JAXWSUtil.java:412)
at com.sap.engine.services.webservices.espbase.client.bindings.impl.SOAPTransportBinding.call_SOAP(SOAPTransportBinding.java:1421)
at com.sap.engine.services.webservices.espbase.client.bindings.impl.SOAPTransportBinding.callWOLogging(SOAPTransportBinding.java:998)
at com.sap.engine.services.webservices.espbase.client.bindings.impl.SOAPTransportBinding.call(SOAPTransportBinding.java:952)
at com.sap.engine.services.webservices.espbase.client.jaxws.core.WSInvocationHandler.processTransportBindingCall(WSInvocationHandler.java:168)
at com.sap.engine.services.webservices.espbase.client.jaxws.core.WSInvocationHandler.invokeSEISyncMethod(WSInvocationHandler.java:121)
at com.sap.engine.services.webservices.espbase.client.jaxws.core.WSInvocationHandler.invokeSEIMethod(WSInvocationHandler.java:84)
at com.sap.engine.services.webservices.espbase.client.jaxws.core.WSInvocationHandler.invoke(WSInvocationHandler.java:65)
at com.sun.proxy.$Proxy846.e2EAfwkPushStatus(Unknown Source)
at com.sap.smd.agent.wsclients.serverside.dcc.DCCAgentStatusPush.register(DCCAgentStatusPush.java:256)
at com.sap.smd.SMDEventsManager$2.run(SMDEventsManager.java:243)
at com.sap.smd.server.exec.TaskRunner.run(TaskRunner.java:47)
at com.sap.smd.server.util.concurrent.PooledExecutor$Worker.run(PooledExecutor.java:785)
at java.lang.Thread.run(Thread.java:763)
[DCCAgentStatusPush.unregister] Error to push agent event myAgentsHostName
[EXCEPTION]
javax.xml.ws.WebServiceException: Connection IO Exception. Check nested exception for details. (Peer certificate rejected by ChainVerifier).
        at com.sap.engine.services.webservices.espbase.client.jaxws.core.WSInvocationHandler.processTransportBindingCall(WSInvocationHandler.java:174)
        at com.sap.engine.services.webservices.espbase.client.jaxws.core.WSInvocationHandler.invokeSEISyncMethod(WSInvocationHandler.java:121)
        at com.sap.engine.services.webservices.espbase.client.jaxws.core.WSInvocationHandler.invokeSEIMethod(WSInvocationHandler.java:84)
        at com.sap.engine.services.webservices.espbase.client.jaxws.core.WSInvocationHandler.invoke(WSInvocationHandler.java:65)
        at com.sun.proxy.$Proxy512.e2EAfwkPushStatus(Unknown Source)
        at com.sap.smd.agent.wsclients.serverside.dcc.DCCAgentStatusPush.unregister(DCCAgentStatusPush.java:284)
        at com.sap.smd.SMDEventsManager$3.run(SMDEventsManager.java:313)
        at com.sap.smd.server.exec.TaskRunner.run(TaskRunner.java:47)
        at com.sap.smd.server.util.concurrent.PooledExecutor$Worker.run(PooledExecutor.java:785)
        at java.lang.Thread.run(Thread.java:763)
Caused by: com.sap.engine.services.webservices.espbase.client.bindings.exceptions.TransportBindingException: Connection IO Exception. Check nested exception for details. (Peer certificate rejected by ChainVerifier).
        at com.sap.engine.services.webservices.espbase.client.bindings.impl.SOAPTransportBinding.outputSOAPMessage(SOAPTransportBinding.java:426)
        at com.sap.engine.services.webservices.espbase.client.bindings.impl.SOAPTransportBinding.call_SOAP(SOAPTransportBinding.java:1371)
        at com.sap.engine.services.webservices.espbase.client.bindings.impl.SOAPTransportBinding.callWOLogging(SOAPTransportBinding.java:997)
        at com.sap.engine.services.webservices.espbase.client.bindings.impl.SOAPTransportBinding.call(SOAPTransportBinding.java:951)
        at com.sap.engine.services.webservices.espbase.client.jaxws.core.WSInvocationHandler.processTransportBindingCall(WSInvocationHandler.java:168)
        ... 9 more
Caused by: iaik.security.ssl.SSLCertificateException: Peer certificate rejected by ChainVerifier
        at iaik.security.ssl.y.a(SourceFile:932)

Solution

  • Enable debug logging for the logging location "com.sap.smd.server" and search the traces for "DCCAgentStatusPush" in order to get more detailed information.
  • In the Agent Administration, go to Application Configuration → "com.sap.smd.agent.application.global.configuration" and verify that the values of the following properties are correct: 
    • dcc.url
    • e2e.maiIntern.user 
    • e2e.maiIntern.password
  • On the Solution Manager AS ABAP, open the transaction /SOAMANAGER, go to "Web Service Configuration" and search for the object name "E2E_DCC_PUSH". Open the Details and click on the glasses icon of the "binding" entry. Check the authentication settings in the security tab.
  • On the Solution Manager AS ABAP, open the transaction /STRUST and check the certificates.

DPC Push Fails

If the DPC push fails, the following is logged in the SMDSystem.*.log:

[DPCServicePushMetricJob.pushSimpleEvents] Error occurred when calling the DPC Push web service. (Endpoint: http://solutionManagerHostName:8000/sap/bc/srt/scs/sap/e2e_dpc_push?sap-client=100).
...
Caused by: java.net.SocketTimeoutException: connect timed out (local port 12345 to address 123.45.67.89(agentsHostName.domain.name), remote port 8000 to address 10.11.12.13(solutionManagerHostName.domain.name))

In this case, ensure that the host name (here solutionManagerHostName) and the port (here 8000) are correct (the local port number (here 12345) does not matter).

If the host name or the port number is not correct, go to the Solution Manager Configuration → Infrastructure Preparation → 2. Set up Java Connectivity → 2.1 Define HTTP Connectivity and verify the ABAP Application Server settings. The connection can be tested by clicking on Test Connectivity. 

If the host name and the port number that are used by the DPC push are correct, verify that the connection is not blocked by a firewall. Open the Agent Administration → Advanced Settings → Diagnostics Agent Support Tool → Agent OS Command. Select the affected agent and execute the following command; adjust the host name and the port number accordingly. Mind that the execution might take up to a minute. 

telnet solutionManagerHostName 8000

If the output contains the following, there might be an issue with your firewall:

Trying 123.45.67.89...
telnet: connect to address 123.45.67.89: Connection refused


A positive result of the test would provide an output as follows:

ERROR - [ telnet hostname.com 44300 ] - hostname=[hostname.com] 
Detailed Information - Failed to execute unspecified - Return code: 143
Trying 123.45.67.89...
Connected to solutionManagerHostName.
Escape character is '^]'.

Alternatively, the following commands could be used to verify that a dedicated port is reachable:

cat < /dev/tcp/hostname.com/50204
nc -zv hostname.com 50204

DPC Push Fails due to an SSL Handshake Failure

If DPC Push fails due to the following exception, enable SSL Logging on the Agent Side and check the logs for details.

Error [DPCServicePushMetricJob.pushSimpleEvents] Error occurred when calling the DPC Push web service. (Endpoint: https://my-host-name.com/sap/bc/srt/scs/sap/e2e_dpc_push?sap-client=100).
[EXCEPTION]
java.lang.reflect.UndeclaredThrowableException
at com.sun.proxy.$Proxy494.e2EDpcPushMetrics(Unknown Source)
at com.sap.smd.dpc.job.DPCServicePushMetricJob.pushSimpleEvents(DPCServicePushMetricJob.java:132)
at com.sap.smd.dpc.job.DPCServicePushMetricJob.run(DPCServicePushMetricJob.java:69)
at com.sap.smd.server.exec.TaskRunner.run(TaskRunner.java:47)
at com.sap.smd.server.util.concurrent.PooledExecutor$Worker.run(PooledExecutor.java:785)
at java.lang.Thread.run(Thread.java:743)
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.GeneratedMethodAccessor5774.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at com.sap.smd.api.util.SynchronizedProxy$SyncHandler.invoke(SynchronizedProxy.java:27)
... 6 more
Caused by: com.sun.xml.internal.ws.client.ClientTransportException: HTTP transport error: javax.net.ssl.SSLHandshakeException: Received fatal alert: handshake_failure


JMS Scheduler - Missing Permissions for user J2EE_GUEST

The following exception occurs when the J2EE_GUEST user has no permission to perform the needed actions using the JMS scheduler queue.

javax.jms.JMSSecurityException: User: J2EE_GUEST has not permission: vpName: default, type: queue, action: browse, destination: SchedulerQueue
 at com.sap.jms.server.sc.UMESecurityProvider.checkPermission(UMESecurityProvider.java:223)
 at com.sap.jms.server.sc.UMESecurityProvider.checkDestinationBrowsePermission(UMESecurityProvider.java:126)
 at com.sap.jms.server.JMSVirtualProviderProcessor.consumerCreate(JMSVirtualProviderProcessor.java:352)
 at com.sap.jms.client.session.JMSSession.createBrowser(JMSSession.java:368)
 at com.sap.jms.client.session.JMSSession.createBrowser(JMSSession.java:341)
 at com.sap.sup.admin.scheduler.JmsScheduler.checkAndSendTasksToJms(JmsScheduler.java:1175)
 at com.sap.sup.admin.scheduler.JmsScheduler.access$000(JmsScheduler.java:84)
 at com.sap.sup.admin.scheduler.JmsScheduler$1.run(JmsScheduler.java:976)
 at com.sap.engine.core.thread.impl3.ActionObject.run(ActionObject.java:37)
 at java.security.AccessController.doPrivileged(Native Method)
 at com.sap.engine.core.thread.impl3.SingleThread.execute(SingleThread.java:185)
 at com.sap.engine.core.thread.impl3.SingleThread.run(SingleThread.java:302)

Recommended Solution

Update your LM-SERVICE to SP 6 or higher.

Alternative Solution

Ignore the error mes.sage as it does no crucial functionality is impaired.

Alternative Solution (Not Recommended)

Add the required permissions to the J2EE_GUEST user:

  1. Create a new role with a unique name possibly "JMS".

  2. In the assigned user choose J2EE_GUEST.
  3. In the assigned actions tab search for*jms*
  4. Choose "Service / Application: jms.default", "Name": "queue.all.all".
  5. Save the changes made.

Agent Authentication and Connectivity - Appendix

Connection Types

The Diagnostics Agent can connect to the Solution Manager via various protocols. The most commonly used combinations can be examine and modified via the Agent Administration's "Agent Connectivity" tab. If you need to change the connection settings of an agent that is currently not connected to the Solution Manager AS Java, you cannot use this UI. Instead, use the smdsetup script or manually edit the agent's runtime.properties as shown in the table below. The connection settings shown in the Agent Connectivity tab match the connection URLs specified in the runtime.properties. Use the table below with "myhost" being the host name of the Solution Manager AS Java, 1234 being the port number of the P4(SSL), and 5678 the port number of the Message Server HTTP(S) port.

Protocol
"smdsetup" Script Command
smd.agent.connection.url
Common Port Numbers
Direct P4 p4\://myhost\:12345<instance number>04
Direct P4 SSL p4s\://myhost\:12345<instance number>05
MS / P4managingconf hostname:"sapms://myhost" port:"5678"ms\://myhost\:5678/P48102
MS / P4 SSL ms\://myhost\:5678/P4S8102
MS-HTTPS / P4 mshttps\://myhost\:5678/P48103, 44402, or 44403
MS-HTTPS / P4 SSL mshttps\://myhost\:5678/P4S8103, 44402, or 44403

Using the Connection Log

To investigate connectivity issues, it is very helpful to use the Connection log of the Agent Administration. To enable it, open the Connection Log of the Agent Administration end adjust the settings as shown below.


P4 Logging on the Agent Side

  1. Stop the Diagnostics Agent (e.g. by executing  stopsap as  <SID>adm).
  2. Add the following snippet to the XML element log-destinations in /usr/sap/<SID>/SMDA<instance nr>/SMDAgent/configuration/log-configuration.xml.

    <log-destination count="10" effective-severity="All" limit="1000000" name="p4monitor" pattern="./log/p4monitoring.log" type="FileLog">
      <formatter-ref name="trc"/>
    </log-destination>
    <log-destination count="10" effective-severity="All" limit="1000000" name="p4" pattern="./log/p4.log" type="FileLog">
      <formatter-ref name="trc"/>
    </log-destination>
  3. Additionally, add the snippet below to the XML element log-controllers and save the file:

    <log-controller effective-severity="All" name="com.sap.engine.services.rmi_p4">
      <associated-destinations>
         <destination-ref association-type="LOG" name="SMDAGENT_ALL_TRACE_DESTINATION"/>
         <destination-ref association-type="LOG" name="p4"/>
      </associated-destinations>
    </log-controller>
    <log-controller effective-severity="All" name="com.sap.smd.agent.rmi_p4.monitor">
      <associated-destinations>
        <destination-ref association-type="LOG" name="p4monitor"/>
        <destination-ref association-type="LOG" name="SMDAGENT_ALL_TRACE_DESTINATION"/>
      </associated-destinations>
    </log-controller> 
  4. Start the Diagnostics Agent (e.g. by executing startsap as <SID>adm).

  5. All P4-related logs will be written to /usr/sap/<SID>/SMDA<instance nr>/SMDAgent/log/p4.*.log  or rather p4monitoring.log.

JSSE SSL Logging on the Agent Side

To enable SSL debug logging for connections that are established by the Java's default SSL engine JSSE, do the following. Although JSSE is used for most of the SSL connections, there are collectors (e.g. the SCC Collector (SAP Cloud Connector Collector) and the SAPPingHTTPCollector) that are instead using the IAIK library. The following configuration will only generate logs for SSL connections that are established by the JSSE engine. There is no way to get debug logs for connections established by the IAIK library.

  1. Stop the Diagnostics Agent (e.g. by executing  stopsap as  <SID>adm).
  2. Add the Java parameter -Djavax.net.debug=all to smdagent.javaParameters in /usr/sap/<SID>/SMDA<instance nr>/SMDAgent/smdagent.properties.
    If 
    usr/sap/<SID>/SMDA<instance nr>/smdagentgroup.properties exists (if Agents-on-the-Fly is enabled), you need to add the parameter in this file.

  3. Remove /usr/sap/<SID>/SMDA<instance nr>/profile/smd.properties.vmprop if it exists.

  4. Start the Diagnostics Agent (e.g. by executing startsap as <SID>adm).

  5. All SSL-related logs will be written to /usr/sap/<SID>/SMDA<instance nr>/work/jvm_smdagent.out.

Enable Server-Side Debug Logging 

In the NetWeaver Administrator go to Troubleshooting → Logs and Traces → Log Configuration. Switch to the Tracing Locations view and enter a package (e.g. com.sap.smd.*) or a class name (e.g. com.sap.smd.SMDManager), click on "Go" and select the desired location. Change the severity to Debug and click on "Copy to Subtree" in order to make the configuration effective for all class in packages. Afterwards click on "Save Configuration".

Mind that locations which have never logged anything before are not visible and cannot be selected. If you want to change the logging configuration for those locations, change the configuration for the closest parent and wait for its first log entry. You can now reset the configuration for the parent and change it for the dedicated location.

The logging configuration of a dedicated location can be reset to default by selecting the location and clicking on "Reset Location". Do not forget to also click on "Copy to Subtree" (if applicable) before saving the configuration. 

For general connectivity issues it is recommended to set the following locations to Debug:

  • com.sap.smd.server
  • com.sap.smd.SMDManager
  • com.sap.smd.SMDServerHandle

For P4-related issues, additionally increase the log level of:

  • com.sap.engine.services.rmi_p4
  • com.sap.engine.services.cross

Enable and Pull the NetWeaver Security Logs

  1. Open the Netweaver Administrator via  https://<j2ee_host>:<port>/nwa .
  2. Go to Troubleshooting → Logs and Traces → Security Troubleshooting Wizard.
  3. Click on Start Diagnostics
  4. Re-execute the operation you want to examine. E.g., if you want to examine the registration procedure of the Diagnostics Agent, re-start the agent and wait 1-3 minutes.
  5. Afterwards, click on Stop Diagnostics.
  6. Click on Download Zip Archive or rather directly view the logs in your browser.
     

ICM Logs of the Java Server 

The ICM logs can be found at  /usr/sap/<SID>/J<instance number>/work/dev_icm.

To increase the log level go to /usr/sap/<SID>/J<instance number>/exe and execute "icmon pf=/usr/sap/<SID>/J<instance number>/SYS/profile/<SID>_J<instance number>_<HOST>". The icmon program is interactive, this is to increase the trace level enter "+" and press enter. To quit enter "q" followed by enter. For further details see SAP Note 1095475.

ICM Logs of the ABAP Server (SMICM)

Open the SAP Gui transaction SMICM. Click on Goto → Trace File → Display All or rather Save Locally to view or rather save the logs. Click on Goto → Trace Level → Set/Increase/Decrease or rather Default to change the trace level or rather reset it. 

Solution Manager Server Thread Dumps

  1. Open a browser and access NetWeaver Administrator at http://<AS Java Hostname>:<HTTP Port>/nwa
  2. Navigate to Troubleshooting → Advanced Troubleshooting → Thread Dump Analysis.
  3. Click on "Generate Thread Dump", select "All Server Processes", click "OK".
  4. Download the generated thread dump.

Solution Manager Server Heap Dumps

In case of an out-of-memory error, the heap dumps of the Solution Manager are stored as  /usr/sap/<SID>/J<instance number>/j2ee/cluster/server<node number>/*.hprof.

Profiling the Solution Manager AS Java

In oder to analyse resource consumption issues of the Solution Manager AS Java (or any other SAP JVM process), trigger an automatic thread dump generation and import it to the SAP JVM Profiler or send it to the SAP support if requested. Furthermore, the SAP JVM Profiler can be used to trigger a Performance Hotspot Analysis.

Debugging the Solution Manager AS Java

To enable the debug mode of an AS Java, open the NetWeaver Administrator and go to Operations → Systems → Start & Stop → Java Instances. Select an instance an click on "Enable Debug" in the Java Processes tab. The port number will be displayed.

Using the Java Profiler

In rare cases you might be asked by the SAP support to use the Java Profiler in order to analyse very special performance issue within the Solution Manager AS Java. If so, do the following: 

  1. Install the Java Profiler
  2. On the Solution Manager AS Java host, start the daemon on the remote host by executing "./jvmmond" at sapjvm/bin/ of the AS Java.
  3. Prepare to reproduce the issue you have been asked to reproduce by the SAP support.
  4. Execute the Performance Hotspot Analysis
  5. Reproduce the issue.
  6. Click on "Stop analysis" once the issue has been covered.
  7. Export the result as an *.snp file and send it to the SAP support.
Method Parameter Analysis of Agent Connectivity-Related Issues
  1. Execute the “Method Parameter Analysis”

  2. Import the connectivity-related *.spf file as shown below:

  3. Start the analysis and reproduce the issue.
  4. Stop the analysis once after the issue has been reproduced and attach the exported analysis *.snp file.

Supported Web Dispatcher Versions

See SAP Note  2248724.

  • No labels

2 Comments

  1. wily introscope did not run on solman 7.2. on linux-on-power ? 

    1. This depends on the Introscope version. The current version (10.5) does not run on PowerLinux. Also see Re: RCA_Introscope_Home