Skip to end of metadata
Go to start of metadata

This article describes how to troubleshoot the Diagnostics Agent and provides a brief overview of the most common know issues and how to solve these. In case of Outside Discovery issues go to the section Outside Discovery. In case of connectivity issues between the Diagnostics Agent and the Solution Manager see section Connectivity. If the Diagnostics Agent is not starting at all or has a high CPU or memory consumption go to the section Troubleshooting Booting and Ressource Consumption Issues of the Diagnostics Agent.

Table of Contents

 

 

 

 

Outside Discovery - Troubleshooting

For troubleshooting the Outside Discovery identify the issue and proceed accordingly:

IssueSolution
A logical hostname is alternately associated to two or more physical hosts, i.e. it is toggling between multiple physical hosts.See Hostname Switching.
The Outside Discovery is not sending any data at all.Aquire the Outside Discovery Logs and Analyse them.
The physical or a logical host name reported by the Host Outside Discovery is wrong.Aquire the Outside Discovery Logs and Analyse them.
The FQDN of a Logical Host is Missing.See The FQDN of a Logical Host is Missing.
The host name a database instance is associated to is wrong.Aquire the Outside Discovery Logs and see The Host Name Associated to a Database is Wrong.
Any other issue.Aquire the Outside Discovery Logs and Analyse them. If this did not help, check Outside Discovery - Solution Manager.

Acquiring the Log Files

For troubleshooting the Outside Discovery at the Diagnostics Agent, check the debug log of the Outside Discovery. The logs can be downloaded as described in the SAP Note 1076573. Compendium:

  1. In the Agent Administration application, tab Agent Log Viewer, link Log Configuration, set the levels of the log files e2edcc.log, e2edcc_host.log and e2edcc_db for the Diagnostics Agent on the physical host with the issue to Debug. In case of issues with an agent node (on a logical host), you need to identify the related physical host and change the log configuration of this host). Press Save in the Log Configuration dialog.
  2. Restart the Diagnostics Agent and the SAP Host Agent.
  3. Wait for 1/2 hour.
  4. Again under Agent Log Viewer, use the link Download logs for downloading the logs of the Diagnostics Agent. 
  5. Again under Agent Log Viewer, reset the log levels to their previous values.

For troubleshooting the Outside Discovery at the Solution Manager, do the following:

  1. In the NetWeaver Administrator, navigate to Troubleshooting → Logs and Traces → Log Configuration.
  2. At the ""Tracing Location" and search for "com.sap.sup.dataenrichment.impl.DataEnrichmentManager".

  3. Change the Severity to Debug and save the configuration.

  4. Restart the affected Diagnostics Agent to retrigger the Outside Discovery.
  5. Navigate to Troubleshooting → Logs and Traces → Log Viewer.
  6. Switch to View → Open View → Developer Traces and filter the location for DataEnrichmentManager.

Analysing the Log Files

As the Outside Discovery is only executed on Diagnostics Agent nodes associated to a physical host, the log files located at SMDAgent_<logicalHost>/log (with <logicalHost> being a logical hostname) can be ignored. Identify the most recent SMDAgent/log/e2edcc.*.log file; it should contain Debug log entries. Check the log for the following entries and proceed accordingly:

Log EntrySolution

java.net.SocketTimeoutException: Read timed out (local port 12345 to address 10.11.12.13, remote port 1128 to address 10.11.12.13)

See section The Diagnostics Agent Cannot Reach the Host Agent
Exception calling SAPHostControl Web Service
[DCCService.init] DCC DISABLEDSee section Outside Discovery is Disabled.
[PhysicalHostPushJob] Could not find hostname in 'Hostnames' list. Using Name as fallback full hostnameSee section SAP Host Agent Does not Provide FQDN.

Debug [PhysicalHostPushJob:getComputerSystem] resolveFQDNandIP failed - Using fallback method.
Debug [PhysicalHostPushJob:getComputerSystem] GetFirstValidFQDNandIP - Fallback Method. 

Error [DatabaseInfoPushJob] Exception calling DataEnrichment Manager Bean.
An exception occured during the execution of the function 'FM_DIAGLS_PUSH_DATABASE': The Host 'myHostName' does not existnullnullnull

See section The Host Does not Exist.

Switch to the SAP Host Agent Outside Discovery

In case the Solution Manager Version equals or is higher than 7.2 SP 6 it is recommended to switch to the Outside Discovery provided by SAP Host Agent. This will disable the Outside Discovery functionality of the Diagnostic Agents. See Note 2556432.

The SAP Host Agent will write a dedicated Outside Discovery log file (outsidediscovery.log) which can be found in the SAP Host Agent work directory (/usr/sap/hostctrl/work or C:\Program Files\SAP\hostctrl\work).

In case of problems with the SAP Host Agent Outside Discovery it is recommended to increase the SAP Host Agent trace level to 3 according to the Host Agent's troubleshooting guide. SAP Host Agent's troubleshooting guide. With the increased traces the Outside Discovery should be triggered manually by executing:

/usr/sap/hostctrl/exe/saphostctrl -function ExecuteOutsideDiscovery -sldreg

or

C:\Program Files\SAP\hostctrl\exe\saphostctrl.exe -function ExecuteOutsideDiscovery -sldreg

After the execution all of the SAP Host Agent's log files in the SAP Host Agent work directory should be attached to the incident and send to SAP.

Outside Discovery - Common Issues 

Hostname Switching 

An association of a logical host that keeps switching between multiple physical hosts in the LMDB is most likely caused by a wrong network setup (see SAP Note 2513388)The logical host is switching because it is defined on various physical hosts. Thus, this logical host is independently reported by the Outside Discovery of each physical host it is associated to. As this is not allowed according to SAP Note 962955, you need to fix the setup of the dedicated managed systems such that all logical hosts are unique.

The Outside Discovery only reports logical hosts that are associated to

  1. an SAP System Instance, or
  2. an Agent Node (Agents-on-the-Fly).

As a consequence:

  1. An SAP System Instances must not have be associated to a logical host name that is also associated to an SAP System Instance or an Agent Node on another physical host. This association is declared in the profiles of the SAP System Instances. 

  2. An Agent Node must not be associated to a logical host name that is also associated to an Agent Node or an SAP System Instance on another physical host.

Solution:

 Decide which physical host you want to have the logical host associated to. Remove it on the other physical host as described below:

  1. Ensure that SAP System Instances on different physical hosts have different logical host names as claimed in the NetWeaver installation guide [1] and do not setup Agent Nodes with the same logical host name on a different physical host.
    N.B.: The Outside Discovery asks the SAP Host Agent for the logical hosts of the SAP System Instances on this host and reports these to the LMDB as logical hosts, not checking whether there is actually such a logical host in the network configuration. The Solution Manager processing the Outside Discovery data then apparently stores these as logical hosts in the LMDB, even if the respective host name is not among the “alternative host names”. To change the logical host name of an SAP Instance follow the instructions of the guide "SM71SP5OutsideDiscMissingFQDN" → section "4.3 Step 3 – Checking SAP Profile" available at SAP Note 1611483.
  2. Remove the logical host name in the network configuration or exclude the logical host name in the Agents-on-the-fly setup for one of that physical host. If the two physical hosts are thought to be a high availability setup according to High Availability Installation Strategy (System Switch-Over) keep in mind that the fallback host must only be created once the other host crashed (by a switch over software) [1].

If non of the above mentioned solutions is feasible for you, upgrade your LM-SERVICE.SCA to SP 6 (SAP Note 2507007) or higher and switch to the Host Agent's Outside Discovery. The Outside Discovery of the SAP Host Agent works differently to the Diagnostics Agent, but here is no guarantee that this will fix your issue. The result is depending on your concrete scenario. In case the switch changed something contrary to your expectations, you can always switch back to the Outside Discovery of the Diagnostics Agent.

References:

[1] NetWeaver installation guide: http://service.sap.com/installnw75 (alternative link: https://websmp208.sap-ag.de/guidefinder) section 3.7 Planning the Switchover Cluster for High Availability

Outside Discovery is Disabled

If there is [DCCService.init] DCC DISABLED entries in the SMDAgent/log/e2edcc.*.log files, the Outside Discovery is disabled. To enable the Outside Discovery open the Agent Administration, go to the Application Configuration tab, click on com.sap.smd.agent.application.e2edcc and set e2edcc.enable to true.

SAP Host Agent Does not Provide FQDN of the Physical Host

If there is one of the following entries in the SMDAgent/log/e2edcc.*.log files, you need to review your network configuration, as described in SAP Note 962955

[PhysicalHostPushJob] Could not find hostname in 'Hostnames' list. Using Name as fallback full hostname
Debug [PhysicalHostPushJob:getComputerSystem] resolveFQDNandIP failed - Using fallback method.
Debug [PhysicalHostPushJob:getComputerSystem] GetFirstValidFQDNandIP - Fallback Method. 

 

The FQDN of a Logical Host is Incorrect or Missing

Potential Cause 1

Search the log files for the following:

[PhysicalHostPushJob] 3 SAP instance(s) found.
[PhysicalHostPushJob] SAP instance 1:
[PhysicalHostPushJob]   FQDN          = my-host-name.domain.de
[PhysicalHostPushJob]   hostname      = my-host-name
[PhysicalHostPushJob]   IP address    = 10.10.10.11
[PhysicalHostPushJob]   SID           = XYZ
[PhysicalHostPushJob]   System number = 00
...
[PhysicalHostPushJob] adding agent node: AgentHostname [mm_isAgentNode =true, mm_saplocalhost=my-host-name, mm_resolvedFQN=my-host-name.wrong-domain.de, mm_resolvedSQN=my-host-name, mm_resolvedIP=10.10.10.11, mm_sid=SMD, mm_pathInstance=/usr/sap/SMD/SMDA98, mm_instanceNumber=98]

Here the FQDN of the SAP instance on my-host-name is correct (my-host-name.domain.de), but the LMDB shows a different FQDN, namely my-host-name.wrong-domain.de, because the FQDN of the related Diagnostics Agent node resolves to my-host-name.wrong-domain.de. To fix this, ensure that the host name can be resolved correctly by adjusting your network settings. The Outside Discovery always prefers the FQDNs and IP addresses of the Diagnostics Agent nodes to the information provided by the SAP instances, i.e. the "adding agent node" entry needs to contain the correct data.

Potential Cause 2

If the FQDN stated in the Outside Discovery log does not contain the domain, the JVM might not be able to resolve the FQDN. The cause might be a misconfigured /etc/Hosts file. An evidence is that, the ping command does not return the correct FQDN, whereas nslookup does:

my-host-name:root> ping -c 1 my-host-name
PING my-host-name: (10.11.12.13): 56 data bytes
64 bytes from 10.11.12.13: icmp_seq=0 ttl=255 time=0 ms
 
my-host-name:root> nslookup my-host-name
Server: 10.11.12.13
Address: 10.11.12.13#42
Name: my-host-name.domain.de
Address: 10.11.12.13

Here, the system identifies the host as "my-host-name" and not with the FQDN. If the operating system can resolve a host name by the hosts file, The native OS functions (and that's what is called by Java) return the first host name that's configured in the hosts-file, namely "my-host-name". An nslookup is skipped then.

To fix this, change the /etc/hosts line:

10.11.12.13 my-host-name my-host-name.domain.de

to

10.11.12.13 my-host-name.domain.de my-host-name

In rare cases the Java parameter "-Dcom.sap.jvm.net.resolveLocalhost=true" might also help.

The Host Name Associated to a Database Does not Exist

Error [DatabaseInfoPushJob] Exception calling DataEnrichment Manager Bean.
[EXCEPTION]
java.rmi.RemoteException: RfcExecutionException; nested exception is: 
	com.sap.sup.admin.abap.rfc.exception.RfcExecutionException: An exception occured during the execution of the function 'FM_DIAGLS_PUSH_DATABASE': The Host 'myHostName' does not existnullnullnull
...

The log files of the Outside Discovery contain the error message stated above, if the Outside Discovery tries to report a database installation to the LMDB which is rejected because the host name that is associated to this database is not know to the LMDB. The database is reachable via myHostName and there is no mechanism to prohibit this. The Database Outside Discovery can only report a database if the "connect address" has been previously reported as a host by the Host Outside Discovery. The latter only reports host names that are either the physical host name the Diagnostics Agent is installed on, a logical host name associated to an SAP System Instance, or a logical host name associated to a Diagnostics Agent node. A host name that is neither associated to a Diagnostics Agent node nor an SAP System Instance is considered as irrelevant an will not be reported to the LMDB.

The following options will fix this issue:

Recommended Solution:

Switch to the Host Agent's Outside Discovery (LM-SERVICE SP 6 (SAP Note 2507007) or higher is required).  

Alternative Solutions:

  1. Reconfigure the managed system such that the host name is reported by the Host Outside Discovery.
  2. If you do not want to reach the database via myHostName, remove the host name from your network configuration.
  3.  Manually create the missing host in the LMDB.

The Host Name Associated to a Database is Wrong

If a database instance is associated to a unintended host name in the LMDB or if the association between a database and the desired host name gets deleted automatically every once in a while, access the Outside Discovery log files as described in section Aquire the Outside Discovery Logs, search for "DatabaseInfoPushJob.getDatabaseList " and detect the dedicated database instance. You will find log entries similar to the following:

[DatabaseInfoPushJob.getDatabaseList]  > Property: [Database/InstanceName]=[DBX].
[DatabaseInfoPushJob.getDatabaseList]  > Property: [Database/Host]=[my-host-name].
[DatabaseInfoPushJob.getDatabaseList]  > Property: [Database/Vendor]=[sap].
[DatabaseInfoPushJob.getDatabaseList]  > Property: [Database/Type]=[sap].
...
[DatabaseInfoPushJob] Get Database Properties for: Database[DBX1][DBX][SAP] .
[DatabaseInfoPushJob] > Property: [Database/DBRelease]=[12.3.4.5.6].
[DatabaseInfoPushJob] > Property: [Database/Capability/CopyMethods]=[Offline,Online].
[DatabaseInfoPushJob] > Property: [Database/ConnectAddress]=[Protocol=TCP;Host=my-connect-host-name;Port=1234 IPC:LISTENER].
[DatabaseInfoPushJob] > Property: [Database/ConnectAddress]=[Protocol=TCP;Host=my-second-connect-host-name;Port=5678 IPC:LISTENER].
[DatabaseInfoPushJob] > Property: [Database/InstanceList]=[Name=DBX1;Host=my-third-host-name;Name=DBX2;Host=my-forth-host-name].
...

The host name the Outside Discovery associates to the database is taken from "Database/Host", here "my-host-name". If this is not the desired host name, adjust the host name of the dedicated database accordingly. Additioally the last "Database/ConnectAddress" entry is also reported to the LMDB (here "my-second-connect-host-name"). 

Further References

 

 

 

 

 

Host Agent Connectivity - Troubleshooting

If the Diagnostics Agent cannot ...

Relevant Logs

  • /usr/sap/hostctrl/work/sapstartsrv.log
  • /usr/sap/DAA/SMDA98/SMDAgent/log/SMDSystem.*.log

It might be helpful to enable debug logging for the locations using the 

  • SAPHostAgentTrustedConnectionConfigurator
  • SMDPlugginLogger
  • com.sap.smd.agent.SMDAgent

Unable to Open Connection to Host

If the Diagnostics Agent cannot resolve the host name associated to the Host Agent (usually this should be the host name of the physical host the Diagnostics Agent is running on), the following exception is logged:

Feb 8, 2019 2:26:20 PM [Thread[ExRun:e2emai:jobmgr_10,5,e2emai:jobmgr:ExecTG]] Error      Error occurred in TaskRunner
[EXCEPTION]
com.sap.smd.mai.model.collector.CollectorException: com.sap.smd.mai.model.collector.CollectorException: [SAPHostControlPortTypeHandler.collect] Web Service execution failed.
	at com.sap.smd.mai.model.collector.SAPHostControlWSCollector.collect(SAPHostControlWSCollector.java:60)
	at com.sap.smd.mai.job.MetricJobRunner.run(MetricJobRunner.java:32)
	at com.sap.smd.server.exec.TaskRunner.run(TaskRunner.java:47)
	at com.sap.smd.server.util.concurrent.PooledExecutor$Worker.run(PooledExecutor.java:785)
	at java.lang.Thread.run(Thread.java:763)
Caused by: com.sap.smd.mai.model.collector.CollectorException: [SAPHostControlPortTypeHandler.collect] Web Service execution failed.
	at com.sap.smd.mai.model.collector.saphostctrl.SAPHostControlPortTypeHandler.collect(SAPHostControlPortTypeHandler.java:167)
	at com.sap.smd.mai.model.collector.SAPHostControlWSCollector.collect(SAPHostControlWSCollector.java:51)
	... 4 more
Caused by: com.sap.smd.agent.plugin.connectors.webservice.WebServiceInvocationException: Webservice invocation error during the trusted connection configuration occured on Binding Provider JAX-WS RI 2.1.6 in JDK 6: Stub for http://myhostname.berlin.de:1128/SAPHostControl.cgi
	at com.sap.smd.agent.plugin.connectors.webservice.JaxWebserviceInvocationHandler.invoke(JaxWebserviceInvocationHandler.java:85)
	at com.sun.proxy.$Proxy28.getDatabaseStatus(Unknown Source)
	at sun.reflect.GeneratedMethodAccessor52.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
	at java.lang.reflect.Method.invoke(Method.java:597)
	at com.sap.smd.agent.facade.hostagent.HostAgentSyncProxy$SyncHandler.singleInvoke(HostAgentSyncProxy.java:116)
	at com.sap.smd.agent.facade.hostagent.HostAgentSyncProxy$SyncHandler.invoke(HostAgentSyncProxy.java:74)
	at com.sun.proxy.$Proxy29.getDatabaseStatus(Unknown Source)
	at com.sap.smd.mai.model.collector.saphostctrl.SAPHostControlPortTypeHandler.retrieveDBStatus(SAPHostControlPortTypeHandler.java:247)
	at com.sap.smd.mai.model.collector.saphostctrl.SAPHostControlPortTypeHandler.collect(SAPHostControlPortTypeHandler.java:157)
	... 5 more
Caused by: com.sap.smdagent.plugins.connectors.webservice.SAPHostAgentCheckVersionException: Cannot establish connection to SAP Host agent webservice. (Cannot check the version of SAP Host agent cause byError to perform http request cause by java.io.IOException: Unable to open connection to host "myhostname.berlin.de:1128". The host is down or unavailable...; nested exception is: 
	java.io.IOException: Unable to open connection to host "myhostname.berlin.de:1128". The host is down or unavailable..)
	at com.sap.smd.agent.plugin.connectors.webservice.SAPHostAgentTrustedConnectionConfigurator.checkWSDLHttpAccess(SAPHostAgentTrustedConnectionConfigurator.java:411)
	at com.sap.smd.agent.plugin.connectors.webservice.SAPHostAgentTrustedConnectionConfigurator.setupTrustedConnection(SAPHostAgentTrustedConnectionConfigurator.java:65)
	at com.sap.smd.agent.plugin.connectors.webservice.JaxWebserviceInvocationHandler.invoke(JaxWebserviceInvocationHandler.java:74)
	... 14 more

Solution:

 

Change the value of SAPLOCALHOST in the runtime.properties of the Diagnostics Agent to "locahost" (recommended) or any other name of the physical host the affected Diagnostics Agent is running on. Ensure that the host name can be resolved on the host itself.

The Diagnostics Agent Cannot Reach the Host Agent

In case of error like

Error
Location com.sap.smd.e2edcc.db
Text [DatabaseInfoPushJob] Exception calling SAPHostControl Web Service.
[EXCEPTION]
com.sap.smd.agent.facade.hostagent.HostAgentNotAvailableException: [HostAgentSyncProxy.SyncHandler.invoke] Time out when waiting for Host Agent to answer an already running process/call.This call is cancelled. Method called: com.sap.smd.agent.wsclients.saphostcontrol.SAPHostControlInterfacelistDatabases.
at com.sap.smd.agent.facade.hostagent.HostAgentSyncProxy$SyncHandler.invoke(HostAgentSyncProxy.java:51)
at com.sun.proxy.$Proxy15.listDatabases(Unknown Source)
at com.sap.smd.agent.plugins.dcc.job.DatabaseInfoPushJob.getDatabaseList(DatabaseInfoPushJob.java:363)
at com.sap.smd.agent.plugins.dcc.job.DatabaseInfoPushJob.run(DatabaseInfoPushJob.java:166)
at com.sap.smd.server.exec.TaskRunner.run(TaskRunner.java:47)
at com.sap.smd.server.util.concurrent.PooledExecutor$Worker.run(PooledExecutor.java:781)
at java.lang.Thread.run(Thread.java:763)


or

Error      [PhysicalHostPushJob] Internal Exception.
[EXCEPTION]
com.sap.smd.agent.plugin.connectors.webservice.WebServiceInvocationException: java.net.SocketTimeoutException: Read timed out (local port 56789 to address 10.11.12.13, remote port 1128 to address 10.11.12.13)
	at com.sap.smd.agent.plugin.connectors.webservice.JaxWebserviceInvocationHandler.invoke(JaxWebserviceInvocationHandler.java:126)
	at com.sun.proxy.$Proxy112.getComputerSystem(Unknown Source)
	at sun.reflect.GeneratedMethodAccessor297.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
	at java.lang.reflect.Method.invoke(Method.java:597)
	at com.sap.smd.agent.facade.hostagent.HostAgentSyncProxy$SyncHandler.singleInvoke(HostAgentSyncProxy.java:116)
	at com.sap.smd.agent.facade.hostagent.HostAgentSyncProxy$SyncHandler.invoke(HostAgentSyncProxy.java:74)
	at com.sun.proxy.$Proxy113.getComputerSystem(Unknown Source)
	at com.sap.smd.agent.plugins.dcc.job.PhysicalHostPushJob.executeOutsideDiscovery(PhysicalHostPushJob.java:519)
	at com.sap.smd.agent.plugins.dcc.job.PhysicalHostPushJob.run(PhysicalHostPushJob.java:301)
	at com.sap.smd.server.exec.TaskRunner.run(TaskRunner.java:47)
	at com.sap.smd.server.util.concurrent.PooledExecutor$Worker.run(PooledExecutor.java:781)
	at java.lang.Thread.run(Thread.java:763)
Caused by: javax.xml.ws.WebServiceException: java.net.SocketTimeoutException: Read timed out (local port 56789 to address 10.11.12.13, remote port 1128 to address 10.11.12.13)
	at com.sun.xml.internal.ws.transport.http.client.HttpClientTransport.readResponseCodeAndMessage(HttpClientTransport.java:201)
	at com.sun.xml.internal.ws.transport.http.client.HttpTransportPipe.process(HttpTransportPipe.java:151)
	at com.sun.xml.internal.ws.transport.http.client.HttpTransportPipe.processRequest(HttpTransportPipe.java:83)
	at com.sun.xml.internal.ws.transport.DeferredTransportPipe.processRequest(DeferredTransportPipe.java:78)
	at com.sun.xml.internal.ws.api.pipe.Fiber.__doRun(Fiber.java:587)
	at com.sun.xml.internal.ws.api.pipe.Fiber._doRun(Fiber.java:546)
	at com.sun.xml.internal.ws.api.pipe.Fiber.doRun(Fiber.java:531)
	at com.sun.xml.internal.ws.api.pipe.Fiber.runSync(Fiber.java:428)
	at com.sun.xml.internal.ws.client.Stub.process(Stub.java:211)
	at com.sun.xml.internal.ws.client.sei.SEIStub.doProcess(SEIStub.java:138)
	at com.sun.xml.internal.ws.client.sei.SyncMethodHandler.invoke(SyncMethodHandler.java:98)
	at com.sun.xml.internal.ws.client.sei.SyncMethodHandler.invoke(SyncMethodHandler.java:78)
	at com.sun.xml.internal.ws.client.sei.SEIStub.invoke(SEIStub.java:110)
	at com.sun.proxy.$Proxy112.getComputerSystem(Unknown Source)
	at sun.reflect.GeneratedMethodAccessor297.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
	at java.lang.reflect.Method.invoke(Method.java:597)
	at com.sap.smd.agent.plugin.connectors.webservice.JaxWebserviceInvocationHandler.invoke(JaxWebserviceInvocationHandler.java:92)
	... 12 more
Caused by: java.net.SocketTimeoutException: Read timed out (local port 56789 to address 10.11.12.13, remote port 1128 to address 10.11.12.13)
	at java.net.SocketInputStream.socketRead0(Native Method)
	at java.net.SocketInputStream.read(SocketInputStream.java:129)
	at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
	at java.io.BufferedInputStream.read1(BufferedInputStream.java:258)
	at java.io.BufferedInputStream.read(BufferedInputStream.java:317)
	at sun.net.www.http.HttpClient.parseHTTPHeader(HttpClient.java:711)
	at sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:654)
	at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1220)
	at java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:379)
	at com.sun.xml.internal.ws.transport.http.client.HttpClientTransport.readResponseCodeAndMessage(HttpClientTransport.java:198)
	... 29 more

see

SAP Host Agent Web Services Are Not Reachable

Furthermore, you could check whether the following URLs are accessible on the affected host. 

http://logicalhostname:1128/SAPHostControl/?wsdl
http://logicalhostname:1128/SAPOscol/?wsdl

If the web services are not available, there is an issue with. See SAP Host Agent Troubleshooting Guide.

 

The Diagnostics Agent's OS User Is not Authorized to Establish a Trusted Connection to the SAP Host Agent

[EXCEPTION]
com.sap.smd.agent.plugin.connectors.webservice.WebServiceInvocationException: Webservice invocation error during the trusted connection configuration occured on Binding Provider JAX-WS RI 2.1.6 in JDK 6: Stub for http://myhost.mydomain.berlin:1128/SAPHostControl.cgi
	at com.sap.smd.agent.plugin.connectors.webservice.JaxWebserviceInvocationHandler.invoke(JaxWebserviceInvocationHandler.java:85)
	at com.sun.proxy.$Proxy51.getComputerSystem(Unknown Source)
	at sun.reflect.GeneratedMethodAccessor43.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
	at java.lang.reflect.Method.invoke(Method.java:597)
	at com.sap.smd.agent.facade.hostagent.HostAgentSyncProxy$SyncHandler.singleInvoke(HostAgentSyncProxy.java:116)
	at com.sap.smd.agent.facade.hostagent.HostAgentSyncProxy$SyncHandler.invoke(HostAgentSyncProxy.java:74)
	at com.sun.proxy.$Proxy52.getComputerSystem(Unknown Source)
	at com.sap.smd.agent.plugins.dcc.job.PhysicalHostPushJob.getComputerSystem(PhysicalHostPushJob.java:1113)
	at com.sap.smd.agent.plugins.dcc.job.PhysicalHostPushJob.executeOutsideDiscovery(PhysicalHostPushJob.java:478)
	at com.sap.smd.agent.plugins.dcc.job.PhysicalHostPushJob.run(PhysicalHostPushJob.java:301)
	at com.sap.smd.server.exec.TaskRunner.run(TaskRunner.java:47)
	at com.sap.smd.server.util.concurrent.PooledExecutor$Worker.run(PooledExecutor.java:781)
	at java.lang.Thread.run(Thread.java:763)
Caused by: com.sap.smdagent.plugins.connectors.webservice.SAPHostAgentAccessDeniedException: Cannot establish connection to SAP Host agent webservice. The diagnostics agent os user xyz\SAPServiceDAA is not authorized to establish a trusted connection to SAP Host agent, check the service/admin_users property of associated profile.)
	at com.sap.smd.agent.plugin.connectors.webservice.SAPHostAgentTrustedConnectionConfigurator.setupTrustedConnection(SAPHostAgentTrustedConnectionConfigurator.java:221)
	at com.sap.smd.agent.plugin.connectors.webservice.JaxWebserviceInvocationHandler.invoke(JaxWebserviceInvocationHandler.java:74)
	... 13 more

Solution:

See SAP Host Agent Troubleshooting GuideAlso see 1862333.

  

 

 

 

Agent Authentication and Connectivity - Troubleshooting

To localize the cause of any connectivity issue, it should be identified as a server or an agent issue. The clearest indicator here is whether or not other agents can be authenticated. The following issues are very common and easy to solve. If non of the issues listed below correspond to your issue or if the provided solution did not solve it, proceed with checking the Login Module. If this does neither help follow Checking the Log Files.

Collection of Common Issues

IssueSolution

No Diagnostics Agent cannot connect. The agents do not even appear in the Non-authenticated Agents tab.

Process the following steps in the order specified:

  1. A Diagnostics Agent Cannot Connect
  2. Server Certificate Verification Fails
No Diagnostics Agent cannot connect. The agents appear in the Non-authenticated Agents tab, but clicking on "Trust Agent" does not work.

Process the following steps in the order specified:

  1. Trust Agent Does Not Work
  2. Invalid Configuration on P4S Port
A single Diagnostics Agent cannot connect. The agent does not even appear in the Non-authenticated Agents tab.

Process the following steps in the order specified:

  1. A Diagnostics Agent Cannot Connect
A single Diagnostics Agent cannot connect. The agent appears in the Non-authenticated Agents tab, but clicking on "Trust Agent" does not work.See Trust Agent Does Not Work.
Diagnostics Agents can connect via P4 or P4S, but not via MS or MSHTTPS.See Agent Cannot Connect via the Message Server.
Diagnostics Agents can connect via MS, but not via MSHTTPS. I.e. Diagnostics Agents cannot verify the Message Server certificate.See Message Server Certificate Is not in TrustedCAs.
All connection problems are related to a dedicated Solution Manager instance only.See Agents Cannot Connect to a Dedicated Instance.
In a landscape with a high number of Diagnostics Agents, the number of connected agents is unstable, i.e. agents keep loosing their connection to the Solution Manager, but manage to reconnect every once in a while.See Connectivity Instability in Huge Landscapes.
In the Action Center of the Agent Administration the following message is shown: "The Diagnostic Agent can no longer be updated automatically and will cease to function. Reinstall it according to note 1999000."See The Diagnostic Agent Can no Longer Be Updated Automatically.
On a Solution Manager with multiple instances with Diagnostics Agents connected via P4S the agent shows as disconnected on at least one instanceSee Agents Have Problems to Connect with Multiple Instances via P4S

Log-In Module

All the issues described in the following subsection can be solved as described in section Reconfigure the Log-In Module.

Check the State of Log-In Module and the Root Certificate

Open the Agent Security tab of the Agent Administration and make sure that the log-in module has been configured, and that the root certificate used for agent authentication is generated and valid. 

Check the Configuration of the Log-In Module (ClientCertLoginModule) 
  1. Check that the log-in module has been configured correctly. In the Netweaver Administrator go to Configuration →  Security → Authentication and Single Sign-On select the component SAP-J2EE-Engine  → com.sap.engine.services.security.server.jaas.ClientCertLoginModule  
    1. In the upgrade scenario it is important that there are two rules for the ClientCertLoginModule (Rule1 is used for the authentication of 7.10 Diagnostics Agents, whereas Rule2 is necessary for 7.20). 
    2. Check the Flag associated to each log-in module. The ClientCertLoginModule should be first, and marked as SUFFICIENT.  No log-in module should be marked as REQUIRED.
SAP-J2EE-Engine

Check that the component SAP-J2EE-Engine is used by the P4 connection. Open the Netweaver Administrator and go to Configuration →  Security → Authentication and Single Sign-OnCheck that the template used for the Policy Configuration "service.naming" is "SAP-J2EE-Engine".

 

Checking the Log Files

Getting the Connection-Related Log Files

In order to troubleshoot any connection issues, the following logs are relevant. They are required to be attached to any ticket on SV-SMG-DIA-SRV-AGT forwarded to the development support.

Diagnostics Agent Logs
    • Download the Diagnostics Agents log files via the Agent Administration → Agent Log Viewer. Select the relevant agent and click on "Download Logs" and click on "No, standard logs only".
    • Alternatively (e.g. if the agent cannot be accessed via the Agent Administration due to connection issues), provide the latest¹ SMDSystem.*log file which is located at /usr/sap/<SID>/SMDA<instance nr>/SMDAgent/log/ on OS level of the relevant agent.
    • More details: https://launchpad.support.sap.com/#/notes/0002455767.
Solution Manager Traces
    • In the NetWeaver Administrator got to Troubleshooting → Logs and Traces → Log Viewer. Click on View → Open View → Developer Traces. Afterwards download the logs via Log Files → Download Log Files. In case the Log Viewer does not work check SAP Note 2506964.
    • Alternatively, take the latest¹ *.trc file from /usr/sap/<SID>/J<instance number>/j2ee/cluster/server<node number>/log on OS level of the Solution Manager server.

¹ in regards to the time stamp, not to the file name

 

Identifying the Issue
The Diagnostics Agent's SMDSystem.log containsSolution

Exception during getInitialContext operation. Cannot establish connection to the remote server. No alive connection. Check state of the server

See No Alive Connection.

[DPCServicePushMetricJob.pushSimpleEvents] Error occurred when calling the DPC Push web service. (Endpoint: http://solutionManagerHostName:8000/sap/bc/srt/scs/sap/e2e_dpc_push?sap-client=100).
...
Caused by: java.net.SocketTimeoutException: connect timed out (local port 12345 to address 123.45.67.89(agentsHostName.domain.name), remote port 8000 to address 10.11.12.13(solutionManagerHostName.domain.name))

See DPC Push Fails.
The Solution Manager's developer trace contains 

ConnectionImpl.setConnectionInfo(ConnectionInfo connInfo) : Cannot get information about connection 09 01 00 00 85 ED 3B 00 Error: not known error code received -10

See No Alive Connection.

com.sap.smd.api.util.SMDAgentCertificateGenerator$NoCAonSslPortException: Unable to read CAs

See Unable to Read CAs.

Agent Authentication and Connectivity - Common Issues

This section describes common issues regarding the agent management and their solutions.

A Diagnostics Agent Cannot Connect

If the Diagnostics Agent is not visible at all in the Agent Administration (neither in the "Connected Agents" tab nor in the "Non-Authenticated Agents" tab), a reconfiguration of the connection parameters might be necessary:

In the file system of the Diagnostics Agent's host go to /usr/sap/<SID>/SMDA<instance nr>/work/ and execute the following command: 

 ./smdsetup managingconf hostname:"sapms://host" port:"port"

The host and port parameters can be found at Solution Manager Workcenter → Infrastructure → Framework → Agent Framework:


Wait a few minutes and check the "Connected Agents" tab nor in the "Non-Authenticated Agents" tab for the agent. If applicable, click on "Trust Agent". If the agent does still not occur in the Agent Administration, ensure that the connection URLs specified in the runtime.properties of the Diagnostics Agent match the pattern described at the section Connection Types.

Trust Agent Does not Work

If not already done, configure the Solution Manager to support the authentication of Diagnostics Agents via certificate as follows:

  1. Make sure that each Solution Manager AS Java instance must not have more than one node. Multi-node instances cause serious agent connectivity issues, e.g. the "Trust Agent" functionality will not work.
  2. If the version of the agent (shown in the "Non-authenticated Agents" tab) and the LM-SERVICE version differ, select the agent and click on Update Agent. Once the agent has been up- or downgraded, click on "Trust Agent(s)".

  3. If this does not help, go to the installation directory of the affected agent node and delete configuration/secstore.properties and configuration/master_password (if existing), set smdserver.connection.requiresAuthentication=verified-certificate in the configuration/runtime.properties and restart the agent to reset all configuration and client certificates and get it back into the non-authenticated mode. Afterwards, authenticate the agent by clicking on "Trust Agent" in the Agent Administration.

  4. Set the TrustedP4S property within NWA. For more information, see SAP Note 2013578.
  5. In the SOLMAN_SETUP transaction, under Infrastructure Preparation, go to step 2.3 ("Diagnostics Agent Authentication"). Click on "Generate new certificate", and "Save".
  6. Restart the AS Java.
  7. Again try to Trust the agent.
  8. If the agent does still not appear in the "Connected Agents" tab, enable the Connection Log. If the Connection Log contains the error message shown below, check that the root certificate for agent authentication has been generated on the server, and note this certificate’s properties. 

  9. Navigate to Configuration → Security → Certificates and Keys and check the key stores TrustedCAs as well as SMDAgentSecurity. If the P4S Status displayed in the Agent Security tab of the Agent Administration is red (error message "The CAs associated with the P4S ports do not match"), go to the server’s NetWeaver Administrator and navigate to Configuration → Security → Certificates and Keys. Verify that the first SMD-CA* certificate of the SMDAgentSecurity key store is valid and equal to one of the SMD-CA* certificate(s) at TrustedCAs (e.g. by comparing the checksums or validity dates). If not, delete or rather re-generate the certificates accordingly. Afterwards, again trust the agent. Whilst SMDAgentSecurity is used during the certificate generation, TrustedCAs is used during the verification.

  10. If the agent does still not appear in the "Connected Agents" tab, navigate to /usr/sap/<SID>/SMDA<Instance-ID>/script on the agent's machine and issue the command 
    smdsetup certificate operation:"LIST_CA_ALIAS"
  11. For each alias name, issue the command
    smdsetup certificate operation:"PRINT"  alias:"p4s_client_cert_<number>"
    and compare the subject name of the certificate to the issuer of the server’s SSL certificates, to be obtained as described in step 5 and check for inconsistencies.
  12. If the agent can still not connect, open the Action Center and create a screen shot of the displayed issues. Furthermore, create screen shots of the TrustedCAs and the SMDAgentSecurity entries in the "Certificates and Keys" view of the NetWeaver Administrator. Enable server-side debug logging and again click on trust agent. Download the Solution Manager AS Java server logs. Provide all the screen shots and the server logs to the SAP support.
  13. If the agent cannot connect after trusting it due to "Exception during getInitialContext operation. Wrong security principal/credentials", check the authentication stack for "service.naming". For this stack, you don't need the logon ticket modules because they are for the HTTP communication while the "service.naming" is for the Java RMI/P4 communication. Therefore, change the "service.naming" stack as follow:

    1. Logon to NetWeaver Administrator and navigate to "Authentication and Single-Sign On".
    2. Find the stack service.naming, you may use the filter Type = Service at the right side.
    3. Go to Edit mode and remove the Used Template → select the blank.
    4. Now, you may remove the unnecessary login modules and leave only the following:
      1. ClientCertLoginModule      SUFFICIENT  (keep the same options)
      2. BasicPasswordLoginModule    REQUISITE
    5. Save the changes and check if it is working. 

Credentials not Found

The following exception in the Diagnostics Agent log indicates that the Diagnostics Agent is still configured to use basic authentication (smdserver.connection.requiresAuthentication in the runtime.properties is case-insensitively set to "true", "yes", or "basic" or it is not set at all), which is not supported anymore. The agent should appear in the "non-authenticated agents" tab. Clicking on "Trust Agent" should make the agent switch to certificate-based authentication (smdserver.connection.requiresAuthentication=verified-certificate). If you still see the exception in the logs of the Diagnostics Agent, the "Trust Agent" operation has not been executed successfully. Figure out why the "Trust Agent" has not been executed successfully by following Trust Agent Does not Work.

 

javax.naming.NoPermissionException: Credentials not found, the configuration is empty.
at com.sap.smd.agent.connection.SMDConnector.checkAuthenticationSettings(SMDConnector.java:1129)
at com.sap.smd.agent.connection.SMDConnectionTask.internalAttemptConnection(SMDConnectionTask.java:537)
at com.sap.smd.agent.connection.SMDConnectionTask$1.run(SMDConnectionTask.java:106)
at com.sap.smd.agent.connection.P4JNDIContextHelper.executeInSecurityContext(P4JNDIContextHelper.java:150)
at com.sap.smd.agent.connection.SMDConnectionTask.attemptConnection(SMDConnectionTask.java:101)
at com.sap.smd.agent.connection.SMDConnectionTask.run(SMDConnectionTask.java:1321)
at java.lang.Thread.run(Thread.java:763)

Server Certificate Issues

Server Certificate Verification Fails

If the agent is not only not connect, but does also not appear in the Non-Authenticated Agents tab, there might be an issue with the server certificate of the P4S port. This can be checked by applying SAP Note 2528155 and setting smd.agent.p4client.checkServerCertificate=false in the agent's runtime.properties and restarting the agent. If the agent can now connect, it was having trouble verifying the server.

Correcting Problems at the P4S Port or rather the Server Certificate 

  1. Modify the server's P4S port to use a certificate that is trusted by the agents. If you have recently modified the certificate, it may be possible to use the previous certificate. The standard certificates delivered with the agents' JVM will also be trusted (including SAPNet, etc.). 
  2. Once the agents can connect again, do the following:
    1. Use the Agent Administration UI to enter the Maintenance Mode.
    2. Go to the Agent Security tab and switch off the verification of the server certificate. 
    3. Switching off the Maintenance Mode will then allow the agents to reconnect and the new property disabling the verification will be communicated to the agents.
    4. Switch on the Maintenance Mode again.
    5. Deploy the new certificate at the P4S port. 
    6. Then switch off the Maintenance Mode, and the agents will reconnect. 
 Invalid Configuration on P4S Port

If the following symptoms apply, there might be an issue with the server certificate associated to the P4S port:

  1. The Diagnostics Agents appear in the "Non-authenticated Agents" tab of the Agent Administration, but clicking on "Trust Agent" does not work for any agent.
  2. The error message "Invalid configuration on P4S port" in the Action Center of the Agent Administration.
  3. The "P4S Status" shown in the "Agent Security" tab is "The CAs associated with the P4S ports do not match".

To (temporarily) solve this issue, apply SAP Note 2528155. You can also use this note to verify issue's cause. As a long-term solution, see 2816888 or fix the SSL setup of your Solution Manager AS Java as described at Preparing the AS Java.

Self-Signed Certificate on the P4S Port

If the Diagnostics Agent cannot connect to the Solution Manager because the "Trust Agent" action failed with the following exception in the Solution Manager AS Java's default traces

SSL port examiner action failed.
[EXCEPTION] com.sap.smd.api.util.SMDAgentCertificateGenerator$SelfSignedCertificateException: ICM_SSL_123456_7890

go to the NetWeaver Administrator → Configuration → Security → SSL and select the SSL Access Point "P4SEC". Ensure that the private key entry "ssl-credentials" is not self-signed, i.e. ensure that more than one certificate is displayed in the "Details" section as shown below:

Incomplete Certificate Chain

Symptom:

  • The "P4S Status" shown in the "Agent Security" tab reads 
    • "The certificate chain associated with the key ssl-credentials in the keystore ICM_SSL_*****_*****" is incomplete"
    • or "The certificate chain associated with the key ssl-credentials in the keystore {0} is incomplete"
  • The traces of the Solution Manager AS Java server contain the following error message:

 

[ServerCAValidityCheck] incomplete chain on P4S port
[EXCEPTION]
com.sap.smd.api.util.SMDAgentCertificateGenerator$IncompleteChainException: ICM_SSL_*****_*****
at com.sap.smd.api.util.SMDAgentCertificateGenerator.getActiveChainFromKeyStore(SMDAgentCertificateGenerator.java:554)
at com.sap.smd.api.util.SMDAgentCertificateGenerator.getCAsFromKeyStore(SMDAgentCertificateGenerator.java:578)
at com.sap.smd.api.util.SMDAgentCertificateGenerator.access$300(SMDAgentCertificateGenerator.java:58)
at com.sap.smd.api.util.SMDAgentCertificateGenerator$SslPortExaminerAction.run(SMDAgentCertificateGenerator.java:445)
at com.sap.smd.api.util.SMDAgentCertificateGenerator$SslPortExaminerAction.run(SMDAgentCertificateGenerator.java:434)
at java.security.AccessController.doPrivileged(Native Method)
at com.sap.smd.api.util.SMDAgentCertificateGenerator.getCertificateIssuers(SMDAgentCertificateGenerator.java:487)
at com.sap.smd.api.util.SMDAgentCertificateGenerator.getActiveSslCertificate(SMDAgentCertificateGenerator.java:428)
at com.sap.smd.server.health.check.ServerCAValidityCheck.doCheck(ServerCAValidityCheck.java:25)
at com.sap.smd.server.health.ServerHealthRunner.run(ServerHealthRunner.java:66)
at com.sap.smd.server.exec.TaskRunner.run(TaskRunner.java:47)
at com.sap.smd.server.util.concurrent.PooledExecutor$Worker.run(PooledExecutor.java:785)
at java.lang.Thread.run(Thread.java:763)


Recommended Solution:
 

Apply SAP Note 2693854

Alternative Solution:

  1. In the NetWeaver Administrator of the Solution Manager AS Java, navigate to Configuration → Security → SSL  and notice the name of the key store view associated to the P4SEC port of the AS Java instance in question. Usually this name starts with "ICM_SSL_".
  2. Navigate to Configuration → Security → Certificates and Keys and selected the mentioned key store view.
  3. Recognize the first entry of this key store view (usually the alias name starts with "SMD-CA_"). This certificate chain is used by the connect Diagnostics Agents to generate their client certificates and to verify the Solution Manager AS Java's identity. Mind the following misconfigurations that might cause the issue:
    1. If there are entries before the first "SMD-CA_" entry or if there is no "SMD-CA_*" entry at all, this is very likely not what you want. Back-up the key store view using the export functionality and remove all entries except the "SMD-CA_" entry you want to be used for the P4S port.
    2. If the alias name of the first entry starts with "SMD-CA_", i.e. the first entry most certainly is the certificate chain you want to be used for the agent management, ensure that the chain of this entry is complete by checking the details section of this entry. Ensure that the last item is a root certificate, that is its subject DN equals its issue DN. Fix the incomplete chain by importing the missing certificate(s) or generate a new chain.

Set Up P4S

If there is no P4S port configured on the Solution Manager AS Java some configuration regarding the certificate generation and authentication might be incomplete. If you are facing any issue connecting an Diagnostics Agent to the Solution Manager, setup SSL and P4S on the Solution Manager AS Java by following 17705852419031, and 2268643. Once the P4S port is available, reconfigure the Diagnostics Agent to connect to the P4S port by adjusting the runtime.properties as described at Connection Types.

Agents Have Problems to Connect with Multiple Instances via P4S

Symptoms

You are running a Solution Manager with multiple instances connecting the Diagnostics Agent via P4S the agent shows as disconnected on at least one instance and as connect on at least one instance. In the Developer Traces of the Solution Manager error messages relating to the P4 connection of the agent should be present.

Reason

The agent only allows one P4S connection of one Solution Manager in some configurations.

Recommended Solution

Upgrade your LM-SERVICE to SP 7 patch 8, SP 8 patch 3 or higher (see SAP Note 2746956).

Work-a-round

Alternatively, go to the Agent Administration and enable the Maintenance Mode. Next, go to Advanced Settings. Look for the parameter single.agent.connection and set it to false. Save and Apply these settings. Go back to the first Tab and disable the Maintenance Mode. Due to the connection errors there seem to be problems to promote the setting change to the agents. If the problems persist for an agent, please restart the agent. If the setting still cannot be promoted to the agent, go to SMDAgent/configuration/runtime.properties and replace "single.agent.connect=true" with "single.agent.connect=false".

Agents Cannot Connect via the Message Server

If a Diagnostics Agent cannot connect to its Solution Manager AS Java via a Message Server, check whether it can connect via a direct P4 or rather P4S connection by adjusting the connection URL in the runtime.properties as follows. Replace

smd.agent.connection.url=ms\://your-solution-manager-java.home\:8101/P4

by

smd.agent.connection.url=p4\://your-solution-manager-java.home\:50004

(adjust the host and domain names as well as the port numbers accordingly) and restart the agent. If the agent can connect via P4 or P4S protocol, but not via MS or MSHTTPS, this indicates a problem with the Message Server. Make sure that the Message Server is correctly configured. If the agent is using MS/HTTPS, examine the output of the URL https:// java-server-host :44402/msgserver/text/logon (adjust the port number as appropriate, 8101 or 8102 for MS/HTTP). Examine the output of this link to assure that the P4 and P4S ports are correct.

Connection via the Message Server Fails due to an UnknownHostException

If the connection via the Message Server fails due to an UnknownHostException similar to the following, adjust the Message Server configuration such that it provides the full qualified host name for the P4(S) port.

URL=ms://myhost.my.domain:8102/P4, protocol=P4(None), Reason: java.net.UnknownHostException: myhost;

Open http://myhost.my.domain:8102/msgserver/text/logon in a browser and check the host name of the P4(S) port. If only the short host name is provided, the Diagnostics Agent might not be able to connect to the P4(S) port. The expected output is as follows:

 

...
P4	myhost.my.domain	50404	LB=1
P4S	myhost.my.domain	50405	LB=1
...

Message Server Certificate Is not in TrustedCAs

If the agent can connect via MS, but not via MSHTTPS and the SMDSystem.*.log contains the error message "Cannot check server certificate on connect: Message server certificate not in TrustedCAs", there is an issue with the server certificate of the Message Server.

Solution

The Diagnostics Agent is only using certificate chains of the key store view associated with the P4SEC port and self-signed certificates of the TrustedCAs key store view to verify the Message Server or rather the ICM. Check the SMDSystem.*.log to identify the certificate the Message Server is using. The Message Server certificate can be a self-signed of the TrustedCAs key store view. Go to Configuration → Security → Certificates and Keys and verify that TrustedCAs key store view contains the message server certificate stated in the SMDSystem.*.log. 
Alternatively, the Message Server can use the signed certificate (not self-signed) of the ICM P4S port. 
In the NetWeaver Administrator, go to Configuration → Security → SSL and ensure that the identified certificate and its complete issuer chain is part of the key store view associated to the P4SEC port of the regarding SAP instance. I.e., ensure that the Message Server is using the same certificate as the ICM P4S port, which must not be self-signed.

Agents Cannot Connect to a Dedicated Instance

On a cluster system, it is important to test the connectivity using a direct P4 or P4S connection to each instance and thereby determine if the problem is only related to one instance. Furthermore ensure that the agents are visible in the Agent Adminstration application on all instances. If the connectivity problem is related to a specific instance, verify in NetWeaver Administrator  Start and Stop Operations that the AS Java Service "smd~service" is started on every instance.

Connectivity Instability in Huge Landscapes 

Each Diagnostics Agent increases the load within the SAP J2EE Engine used as part of the Solution Manager technology stack. When connecting several hundreds of Diagnostics Agents, the below described adjustments are required. Furthermore, it is highly recommended to upgrade to the latest patch level of SP 7 or any higher SP as this version includes some fundamental improvements regarding the connectivity management of Diagnostics Agents. You need also to make sure that each Solution Manager AS Java instance must not have more than one node. Multi-node instances cause serious synchronization issues regarding the connected Diagnostics Agents between the nodes. However, customers who connect more than 1,000 agents often reach a limit of the SAP J2EE Engine used in SAP Solution Manager. As each customer situation is different, SAP cannot exactly specify how many connected agents an individual Solution Manager system can take. SAP in general recommends to either lower the number of Diagnostics Agents (e.g. by dividing the agents to multiple Solution Managers) or to use Focused Run for SAP Solution Manager, which was dedicatedly developed to satisfy advanced customer needs in terms of performance, scalability, security, and automation.

  1. Use the Quick Sizer Tool to calculate CPU, disk, memory and I/O resources for the Solution Manager. 
  2. Access the Agent Administration user interface at https://j2ee-host:port/smd/AgentAdmin.
  3. Switch on the Maintenance Mode.
  4. Change the parameters according to the attached Excel document.

  5. Restart the Solution Manager Java by executing the commands  stopsap  and  startsap ALL  using the user  <SID>adm. To restart individual Java instances you can use the command line tool  jcmon pf=<instance_profile_file>  and within this tool execute the commands 20 and 19 and finally repeat executing the command 1 until all JEE processes are running again.

  6. Turn off the Maintenance Mode. 
During an Update

As the update of the LM-SERVICE component in huge landscapes generates an extra load on the AS Java, it is recommended to restricted the number of connected Diagnostics Agents during an update by introducing the agents tier-wise in batches of 30-50 managed systems (by shutting down the other agents). After the Solution Manager server has updated all agents, the restriction can be dissolved.

Recommended SAP Notes

As reasoned above, it is highly recommended to upgrade to SP 7 in order to stabilise the agent connectivity. If this is not possible, apply the following SAP Notes to reduce the load on the AS Java:

  • SAP Note 2599110 (SP 6) provides lightweight pings from the Diagnostics Agent to the Solution Manager, which reduces the load on the AS Java.
  • SAP Note 2606713 (SP 5 and 6) fixes ping strategy functionality. Until now: A Diagnostics Agent will be disconnected right after a single ping failure. Most likely this would happen if there is already an ongoing issue and therefore putting even more load on the system. The fix will allow tolerating a dedicated number of ping failures before the reconnection procedure is launched.
Analyzing the P4 Thread Usage

In order to analyze the P4 thread usage log in to the Solution Manager Java system and use the "P4 Information Command":

telnet localhost 5<instance number>08
add p4
p4info

If the displayed value of the "Thread usage" stays at 100% for several minutes the connection-related parameters need to be adjusted further.

Multiple IP Addresses for One Solution Manager P4/P4S Port

Running an AS Java with P4/P4S ports associated to more than one IP address many cause issues and introduces instability in the RMI protocol. If you do not have a special purpose, do not maintain multiple IP addresses for the P4 and P4S ports.

Problems are usually indicated by an error message in the Developer Traces in the NWA.

The JNDI lookup or rather the P4 narrow of mediated/*****@1999.01.01-11.11.11.111 did not return within 2000ms


If there are threads hanging when requesting outbound connections from the ICM of the Solution Manager AS Java to a Diagnostics Agent, it is very likely caused by (accidentally) misconfigured P4/P4S IP addresses. Ensure that the P4 or rather the P4S port of each instance is associated to one IP address only. Make sure the IP address you choose is visible by all Diagnostics Agents. Initially, your default profile might read as follows:

...
icm/server_port_0 = PORT=5$$00,PROT=HTTP,TIMEOUT=7200,PROCTIMEOUT=1800
icm/server_port_1 = PORT=5$$04,PROT=P4,TIMEOUT=1800,PROCTIMEOUT=1800
icm/server_port_2 = PORT=5$$06,PROT=P4SEC,TIMEOUT=1800,PROCTIMEOUT=1800
...

To only use one IP address per instance, add the following lines to the instance profile of the affected Solution Manager AS Java instance(s), replacing "123.45.67.89" by the desired IP address. Keep the default profile as it is.

icm/server_port_0 = PROT=HTTP,PORT=5$(SAPSYSTEM)00,TIMEOUT=7200,PROCTIMEOUT=1800
icm/server_port_1 = PROT=P4,PORT=5$(SAPSYSTEM)04,HOST=123.45.67.89,TIMEOUT=1800,PROCTIMEOUT=1800
icm/server_port_2 = PROT=P4SEC,PORT=5$(SAPSYSTEM)06,VCLIENT=0,HOST=123.45.67.89,TIMEOUT=1800,PROCTIMEOUT=1800
icm/server_port_3 = PROT=IIOP,PORT=5$(SAPSYSTEM)07
icm/server_port_4 = PROT=TELNET,PORT=5$(SAPSYSTEM)08,HOST=localhost
icm/server_port_5 = PROT=P4,PORT=5$(SAPSYSTEM)04,HOST=127.0.0.1


Log-In Module Issues

If the log-in module has not been configured correctly, it is very likely that the step 2 .3  Diagnostics Agent Authentication of the Infrastructure Preparation has not been executed successfully. Open the transaction  solman_setup , go to Infrastructure Preparation → 2 . Set Up Java Connectivity  2 .3 Diagnostics Agent Authentication and resolve the issues stated in the logs.

As a work-around the Web Service that configures the log-in module can be triggered manually (This is not officially supported.):

  1. Open the Web Services Navigator (https:// j2ee-host : port /wsnavigator)
  2.  Search for the provider system SMDSetupAuthenticationViImpl.
  3. Select updateLoginModule.
  4. Provide the credentials and execute the Web Service call.

No Alive Connection

If the Diagnostics Agent's SMDSystem.log contains the following error message, 

[EXCEPTION]
javax.naming.NamingException: Exception during getInitialContext operation. Cannot establish connection to the remote server. [Root exception is com.sap.engine.services.rmi_p4.P4IOException: No alive connection. Check state of the server] [...]

The Solution Manager's developer trace contains plenty of errors like:

ConnectionImpl.setConnectionInfo(ConnectionInfo connInfo) : Cannot get information about connection 09 01 00 00 85 ED 3B 00 Error: not known error code received -10

The solution is to restart the Solution Manager Java server.

Unable to Read CAs

The SAP Solution Manager 7.2 default trace shows the following error every 10 minutes:

#2.0##2017 01 31 16:24:06:701#0-200#Error#com.sap.smd.server#
#SV-SMG-DIA-SRV-AGT#tc~smd~server~service#............Thread[ExRun:SMD_MGR_49,5,SMD_MGR:ExecTG]#Plain##
[ServerCAValidityCheck]
[EXCEPTION]
com.sap.smd.api.util.SMDAgentCertificateGenerator$NoCAonSslPortException: Unable to read CAs [...]

This issue can be solved as described in SAP Note 2423083.

Diagnostic Agent Can no Longer Be Updated Automatically

If the error message stated below is shown in the Agent Administration or rather the Action Center a reinstallation of the affected Diagnostics Agents is required. Please proceed as described in SAP Note 1999000. 

The Diagnostic Agent can no longer be updated automatically and will cease to function. Reinstall it according to note 1999000.
False Positive

In very rare cases the error message might persist although the Diagnostics Agent has been updated to version 7.20 without any issues. This case can be distinguished by checking the version of the affected Diagnostics Agent using the "Detailed" view in the Agent Administration. If and only if the error message is shown for an agent having a version ≥ 7.20, the error message is irrelevant and can either be ignored without any consequences or eliminate by proceeding as follows. The root cause for this behaviour is old zombie entries of agents whose server name has changed anytime before. The following procedures describes how to manually delete this odd entries. "Automatically Remove the Zombie Entries" is recommended.

Automatically Remove the Zombie Entries
  1. Open the Agent Administration (transaction sm_workcenter → SAP Solution Manager Administration → Agents Administration → Agent Admin (All Agents))
  2. Click on "Show Offline Agents" in the "Connected Agents" tab.
  3. Click on "Remove Offline Agent Entries".
  4. Wait a few minutes.
  5. Go to SAP Solution Manager Administration → Overview → Landscape → Agent Framework → Status and click on "Refresh" 
  6. The status of the affected agents should have switched from INCOMPATIBLE VERSION to will change to STARTED. 
Manually Remove the Zombie Entries

If you cannot use the "Remove Offline Agent Entries" functionality (e.g. because there are currently disconnected agents you want to keep), you can proceed as follows;

  1. Identify the affected Diagnostics Agent:
    1. In the Agent Administration open the Action Center and remember the Agent ID shown in the "scope" column. E.g.:

      host00_DAA_SMDA98
  2. Start the Config Tool
    1. On Windows, execute

      C:\usr\sap\<SID>\J<Instance>\j2ee\configtool\configtool.bat
    2. On Linux, execute the following. If you are using a remote connection (SSH) remember to enable X11 forwarding.

      cd /usr/sap/<SID>/J<Instance>/j2ee/configtool
      ./configtool.sh
  3. Download the knownAgents file:

    1. In the ConfigTool click Tools → Configuration Editor.

    2. Expand sm_diagnotics_data → smdserver → com.sap.smd.server

    3. Double click knownAgents and download it.

  4. Remove the zombie entries of dedicated agent:
    1. Backup of the knownAgents file and keep.
    2. In the knownAgents file, search for the AgentID of the dedicated agent. 
    3. If there are two or more keys in the properties file having the same Agent ID the root cause described above applies. Please proceed.
      If not, this procedure does not apply to your case and you can stop here. Please create an incident using the component SV-SMG-DIA-SRV-AGT.
    4. Identify and remember all (most likely two) key-prefixs (the part of the key before "~"). In the example below, the prefixes are host00 and host01.

      host00~agentId=host00_DAA_SMDA98
      ...
      host01~agentId=host00_DAA_SMDA98
    5. Search for both key-prefixes and identify the zombi agent entry by looking for the old version (7.10). Here the zombi entry is host00.

      host00~version=7.10.12.5.20150811111924
      ...
      host01~version=7.20.3.0.20160811041533
    6. Delete all lines starting with the key-prefix related to the zombi entry. E.g.:

      host00~ipAddress=123.4.5.6
      host00~lastConnection=Fri Apr 01 12\:42\:00 EDT 2016
      host00~canonicalName=host00
      host00~agentId=host00_DAA_SMDA98
      host00~localName=host00
      host00~serverName=host00
      host00~shortName=host00
      host00~version=7.10.12.5.20150811111924
    7. Save the file.

  5. Upload it to the Solution Manager using the ConfigTool:

    1. Use the small button (tooltip "Switch between view and edit mode") to enable the upload functionality. 

    2. Upload the edited files.

    3. Restart the Java server.

  6. The error message should not be displayed anymore. In case of any issue, you can revert everything by uploading the backup of the original knownAgents file.

Agent Cannot Connect Via SAP Router

The following error in the SMDSystem.*.log indicates that there is an issue with the connection between the Diagnostics Agent and the Solution Manager via an SAP Router.

Connecting to SMD server ms://14.15.16.17:8102/P4 failed - error counter: 1
[EXCEPTION]
javax.naming.NamingException: Exception while trying to get InitialContext. [Root exception is java.io.IOException: Cannot get Socket. Reason:Cannot create NI socket with router string: /H/10.11.12.13/S/3200/H/14.15.16.17/S/8102/P/null]
	[...]
Caused by: java.io.IOException: Cannot get Socket. Reason:Cannot create NI socket with router string: /H/10.11.12.13/S/3200/H/14.15.16.17/S/8102/P/null
	[...]

Assuming that 10.11.12.13 is the IP address and 3200 the port number of the SAP Router and that the Solution Manager is to be reached at 14.15.16.17:57000, search the log for an entry similar to

SAP Router  configured for P4 connection with route /H/10.11.12.13/S/3200.

or rather

SAP Router with password configured for P4 connection with route /H/10.11.12.13/S/3200/H/14.15.16.17/S/57000.

Ensure that the stated SAP Router settings are correct and use the "smdsetup addsaprouter route" and "smdsetup addsaprouter pass" functionallity of the Diagnostics Agent to adjust the SAP Router settings accordingly. The smdsetup script can be found at /usr/sap/DAA/SMDA98/script. For more information see Using the SMD Setup Script and section "7.13 SAP Router" of  Installation of Diagnostics Agent on UNIX and Linux. Furthermore, ensure that the Solution Manager's P4 port (e.g. 50004) is in the SAP Router table.

Afterwards, verify the settings by checking the properties "smd.agent.connection.saprouter" and "smd.agent.connection.transport" at usr/sap/DAA/SMDA98/SMDAgent/configuration/runtime.properties.

Agent Cannot Connect Via P4S Through SAP Router

Solution: Update your LM-SERVICE to a version higher than SP 3 Patch 2, SP 4 Patch 2 or rather SP 5 Patch 0. Also see SAP Note 2458281.

Issues with the Introscope Host Adapter and Byte Code Agent via SAP Router

See https://launchpad.support.sap.com/#/notes/1874044.

Cannot Handle sun.security.x509.X509CertImpl Certificates

While trusting a Diagnostics Agent in the Agent Administration UI, the following error is reported: "java.rmi.RemoteException: Cannot handle certs of class class sun.security.x509.X509CertImpl".

Furthermore, the following entries can be seen in logs of the Diagnostics Agent:

java.rmi.RemoteException: Could not generate certificate; nested exception is: 
java.rmi.RemoteException: Cannot handle certs of class class sun.security.x509.X509CertImpl
at com.sap.smd.agent.connection.configuration.ManagingConnectionManipulator.generateCertificate(ManagingConnectionManipulator.java:1365)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)

java.lang.IllegalArgumentException: argument type mismatch
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)

or

java.lang.IllegalArgumentException: argument type mismatch
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)

To solve this issue, apply SAP Note 2719280. 

Peer certificate rejected by ChainVerifier

The Diagnostics Agent cannot connect to the Solution Manager AS Java due to the following error:

Feb 13, 2019 10:19:06 AM [Thread[Connector,5,main]                     ] Error      Connecting to SMD server mshttps://my.sol.man.sap.com:50002/P4S failed - error counter: 1
[EXCEPTION]
javax.naming.NamingException: Exception while trying to get InitialContext. [Root exception is iaik.security.ssl.SSLCertificateException: Peer certificate rejected by ChainVerifier]
	at com.sap.engine.services.jndi.InitialContextFactoryImpl.getInitialContext(InitialContextFactoryImpl.java:386)
	at javax.naming.spi.NamingManager.getInitialContext(NamingManager.java:684)
	at javax.naming.InitialContext.getDefaultInitCtx(InitialContext.java:313)
	at javax.naming.InitialContext.init(InitialContext.java:244)
	at javax.naming.InitialContext.<init>(InitialContext.java:216)
	at com.sap.smd.agent.connection.SMDConnectionTask.getNewSMDInitialContext(SMDConnectionTask.java:194)
	at com.sap.smd.agent.connection.SMDConnectionTask.getTransientInitialContext(SMDConnectionTask.java:649)
	at com.sap.smd.agent.connection.SMDConnectionTask.internalAttemptConnection(SMDConnectionTask.java:302)
	at com.sap.smd.agent.connection.SMDConnectionTask.access$000(SMDConnectionTask.java:54)
	at com.sap.smd.agent.connection.SMDConnectionTask$1.run(SMDConnectionTask.java:96)
	at com.sap.smd.agent.connection.P4JNDIContextHelper.executeInSecurityContext(P4JNDIContextHelper.java:141)
	at com.sap.smd.agent.connection.SMDConnectionTask.attemptConnection(SMDConnectionTask.java:91)
	at com.sap.smd.agent.connection.SMDConnectionTask.run(SMDConnectionTask.java:1062)
	at java.lang.Thread.run(Thread.java:836)
Caused by: iaik.security.ssl.SSLCertificateException: Peer certificate rejected by ChainVerifier
	at iaik.security.ssl.y.a(SourceFile:932)
	at iaik.security.ssl.n.b(SourceFile:1066)
	at iaik.security.ssl.n.a(SourceFile:1503)
	at iaik.security.ssl.y.d(SourceFile:784)
	at iaik.security.ssl.SSLTransport.startHandshake(SourceFile:569)
	at iaik.security.ssl.SSLTransport.getOutputStream(SourceFile:648)
	at iaik.security.ssl.SSLSocket.getOutputStream(SourceFile:391)
	at com.sap.engine.interfaces.cross.LoadBalancerImpl.staticGetAllAccessPoints(LoadBalancerImpl.java:142)
	at com.sap.engine.interfaces.cross.CrossObjectBroker.getDestination(CrossObjectBroker.java:168)
	at com.sap.engine.services.jndi.InitialContextFactoryImpl.getInitialContext(InitialContextFactoryImpl.java:293)
	... 13 more

Solution:

The Diagnostics Agent might not be able to initially connect to the Solution Manager AS Java via P4S due to missing certificates. Temporarily connect it via P4 using the smdsetup script as described at Trust Agent Does Not Work and click on "Trust Agent". Afterwards switch the connection to the intended type using the Agent Administration's Agent Connectivity tab.

Update List of Trusted CAs

If the Diagnostics Agent are configured to verify the Solution Manager's server certificate (Agent Administration → Agent Security → Server Authentication), it might be necessary to update the agent's trust store (i.e. if the Solution Manager server's certificate changed). This can be done by using  "Update list of trusted CAs" button at the Agent Security tab of the Agent Administration. The CA of the SSL certificate associated with the P4S port and all certificates in the TrustedCA keystore view will be sent to all connected agents. Mind that the button is only available if verification of the SSL certificate is enabled as there is no need to update the trust stores if the agents do not verify the server's certificate.

 

 

Agent Authentication and Connectivity - Diagnostics Agent to Managed System Communication Issues

Diagnostics Agent Cannot Establish a P4 Connection to the Managed System

If the SMDAgentApplication.*.log contains one of the following error messages, there might be an issue with the authentication of the Diagnostics Agent to the managed system's P4 port:

Error      P4 connection error to SAP system [SID/00]
[EXCEPTION]
javax.naming.NoPermissionException: Exception during getInitialContext operation. Wrong security principal/credentials. [Root exception is com.sap.engine.services.security.exceptions.BaseLoginException: Cannot authenticate the user.] [...]
Error      License Management - Unexpected error during service initialization
[EXCEPTION]
com.sap.smdagent.plugins.connectors.p4.exceptions.P4AuthorizationException: Access is denied to SAP System sid [SID/00]: check the connection credentials.More details about the error in agent 'xyz' log file (SMDAgentApplication.X.log).; nested exception is: 
	com.sap.engine.services.security.exceptions.BaseLoginException: Cannot authenticate the user. [...]
License Management - Failed to distribute Licenses to System: SID - The targeted system is stopped [...]

Solution:

The user SM_COLL_SID that is used to access the managed system and collect data is missing some rights. Ensure that user SM_COLL_SID it is not locked. Verify that the user SM_COLL_SID is added to the "Administrators" group in NetWeaver Administrator → Security Provider →  Policy Configurations (select "SAP-J2EE-Engine") → Security Roles → select "Administrators" (details at User Management of the Application Server Java). If the managed system is a double stack, remove SM_COLL_SID from user list and ensure that it is a member of the SAP_J2EE_ADMIN group. If the user SM_COLL_SID is marked as "created manually" at step 6 "Create Users" of the Managed System Configuration, any issue with the user will not be flagged, because you certified that the user is valid and has no issues. 

Further details at Managed System Checklist and Troubleshooting Diagnostics Configuration for Java Systems → Step 2.

Agent Authentication and Connectivity - Java (Solution Manager or Diagnostics Agent) to ABAP Communication Issues

DCC Push Issues

DCC Push is used by the Solution Manager AS Java to send the current connection status of a Diagnostics Agent to the Solution Manager AS ABAP in order to trigger dedicated alerts if applicable. If there is one of the following exceptions in the server traces of the Solution Manager AS Java, the DCC Push functionality is not working. 

[DCCAgentStatusPush.register] Error to push agent event myAgentsHostName
[EXCEPTION]
This exception is wrapper of javax.xml.ws.soap.SOAPFaultException. com.sap.engine.services.webservices.espbase.client.bindings.exceptions.SOAPFaultException: Authorization missing for service "urn:sap-com:document:sap:soap:functions:mc-style E2E_DCC_PUSH", operation "E2eAfwkPushStatus"; more details in the web service error log on provider side (UTC timestamp 20181008134026; Transaction ID 37f693a6da461006bd0fe4e294d08d71)
at com.sap.engine.services.webservices.espbase.client.bindings.impl.JAXWSUtil.processFault(JAXWSUtil.java:412)
at com.sap.engine.services.webservices.espbase.client.bindings.impl.SOAPTransportBinding.call_SOAP(SOAPTransportBinding.java:1421)
at com.sap.engine.services.webservices.espbase.client.bindings.impl.SOAPTransportBinding.callWOLogging(SOAPTransportBinding.java:998)
at com.sap.engine.services.webservices.espbase.client.bindings.impl.SOAPTransportBinding.call(SOAPTransportBinding.java:952)
at com.sap.engine.services.webservices.espbase.client.jaxws.core.WSInvocationHandler.processTransportBindingCall(WSInvocationHandler.java:168)
at com.sap.engine.services.webservices.espbase.client.jaxws.core.WSInvocationHandler.invokeSEISyncMethod(WSInvocationHandler.java:121)
at com.sap.engine.services.webservices.espbase.client.jaxws.core.WSInvocationHandler.invokeSEIMethod(WSInvocationHandler.java:84)
at com.sap.engine.services.webservices.espbase.client.jaxws.core.WSInvocationHandler.invoke(WSInvocationHandler.java:65)
at com.sun.proxy.$Proxy846.e2EAfwkPushStatus(Unknown Source)
at com.sap.smd.agent.wsclients.serverside.dcc.DCCAgentStatusPush.register(DCCAgentStatusPush.java:256)
at com.sap.smd.SMDEventsManager$2.run(SMDEventsManager.java:243)
at com.sap.smd.server.exec.TaskRunner.run(TaskRunner.java:47)
at com.sap.smd.server.util.concurrent.PooledExecutor$Worker.run(PooledExecutor.java:785)
at java.lang.Thread.run(Thread.java:763)
[DCCAgentStatusPush.unregister] Error to push agent event myAgentsHostName
[EXCEPTION]
javax.xml.ws.WebServiceException: Connection IO Exception. Check nested exception for details. (Peer certificate rejected by ChainVerifier).
        at com.sap.engine.services.webservices.espbase.client.jaxws.core.WSInvocationHandler.processTransportBindingCall(WSInvocationHandler.java:174)
        at com.sap.engine.services.webservices.espbase.client.jaxws.core.WSInvocationHandler.invokeSEISyncMethod(WSInvocationHandler.java:121)
        at com.sap.engine.services.webservices.espbase.client.jaxws.core.WSInvocationHandler.invokeSEIMethod(WSInvocationHandler.java:84)
        at com.sap.engine.services.webservices.espbase.client.jaxws.core.WSInvocationHandler.invoke(WSInvocationHandler.java:65)
        at com.sun.proxy.$Proxy512.e2EAfwkPushStatus(Unknown Source)
        at com.sap.smd.agent.wsclients.serverside.dcc.DCCAgentStatusPush.unregister(DCCAgentStatusPush.java:284)
        at com.sap.smd.SMDEventsManager$3.run(SMDEventsManager.java:313)
        at com.sap.smd.server.exec.TaskRunner.run(TaskRunner.java:47)
        at com.sap.smd.server.util.concurrent.PooledExecutor$Worker.run(PooledExecutor.java:785)
        at java.lang.Thread.run(Thread.java:763)
Caused by: com.sap.engine.services.webservices.espbase.client.bindings.exceptions.TransportBindingException: Connection IO Exception. Check nested exception for details. (Peer certificate rejected by ChainVerifier).
        at com.sap.engine.services.webservices.espbase.client.bindings.impl.SOAPTransportBinding.outputSOAPMessage(SOAPTransportBinding.java:426)
        at com.sap.engine.services.webservices.espbase.client.bindings.impl.SOAPTransportBinding.call_SOAP(SOAPTransportBinding.java:1371)
        at com.sap.engine.services.webservices.espbase.client.bindings.impl.SOAPTransportBinding.callWOLogging(SOAPTransportBinding.java:997)
        at com.sap.engine.services.webservices.espbase.client.bindings.impl.SOAPTransportBinding.call(SOAPTransportBinding.java:951)
        at com.sap.engine.services.webservices.espbase.client.jaxws.core.WSInvocationHandler.processTransportBindingCall(WSInvocationHandler.java:168)
        ... 9 more
Caused by: iaik.security.ssl.SSLCertificateException: Peer certificate rejected by ChainVerifier
        at iaik.security.ssl.y.a(SourceFile:932)

Solution:

  • Enable debug logging for the logging location "com.sap.smd.server" and search the traces for "DCCAgentStatusPush" in order to get more detailed information.
  • In the Agent Administration, go to Application Configuration → "com.sap.smd.agent.application.global.configuration" and verify that the values of the following properties are correct: 
    • dcc.url
    • e2e.maiIntern.user 
    • e2e.maiIntern.password
  • On the Solution Manager AS ABAP, open the transaction /SOAMANAGER, go to "Web Service Configuration" and search for the object name "E2E_DCC_PUSH". Open the Details and click on the glasses icon of the "binding" entry. Check the authentication settings in the security tab.
  • On the Solution Manager AS ABAP, open the transaction /STRUST and check the certificates.
DPC Push Fails

If the DPC push fails, the following is logged in the SMDSystem.*.log:

[DPCServicePushMetricJob.pushSimpleEvents] Error occurred when calling the DPC Push web service. (Endpoint: http://solutionManagerHostName:8000/sap/bc/srt/scs/sap/e2e_dpc_push?sap-client=100).
...
Caused by: java.net.SocketTimeoutException: connect timed out (local port 12345 to address 123.45.67.89(agentsHostName.domain.name), remote port 8000 to address 10.11.12.13(solutionManagerHostName.domain.name))

In this case, ensure that the host name (here solutionManagerHostName) and the port (here 8000) are correct (the local port number (here 12345) does not matter).

If the host name or the port number is not correct, go to the Solution Manager Configuration → Infrastructure Preparation → 2. Set up Java Connectivity → 2.1 Define HTTP Connectivity and verify the ABAP Application Server settings. The connection can be tested by clicking on Test Connectivity. 

If the host name and the port number that are used by the DPC push are correct, verify that the connection is not blocked by a firewall. Open the Agent Administration → Advanced Settings → Diagnostics Agent Support Tool → Agent OS Command. Select the affected agent and execute the following command; adjust the host name and the port number accordingly. Mind that the execution might take up to a minute. 

telnet solutionManagerHostName 8000

If the output contains the following, there might be an issue with your firewall:

Trying 123.45.67.89...
telnet: connect to address 123.45.67.89: Connection refused


A positive result of the test would provide an output as follows:

ERROR - [ telnet hostname.com 44300 ] - hostname=[hostname.com] 
Detailed Information - Failed to execute unspecified - Return code: 143
Trying 123.45.67.89...
Connected to solutionManagerHostName.
Escape character is '^]'.

Alternatively, the following commands could be used to verify that a dedicated port is reachable:

cat < /dev/tcp/hostname.com/50204
nc -zv hostname.com 50204
DPC Push Fails due to an SSL Handshake Failure

If DPC Push fails due to the following exception, enable SSL Logging on the Agent Side and check the logs for details.

Error [DPCServicePushMetricJob.pushSimpleEvents] Error occurred when calling the DPC Push web service. (Endpoint: https://my-host-name.com/sap/bc/srt/scs/sap/e2e_dpc_push?sap-client=100).
[EXCEPTION]
java.lang.reflect.UndeclaredThrowableException
at com.sun.proxy.$Proxy494.e2EDpcPushMetrics(Unknown Source)
at com.sap.smd.dpc.job.DPCServicePushMetricJob.pushSimpleEvents(DPCServicePushMetricJob.java:132)
at com.sap.smd.dpc.job.DPCServicePushMetricJob.run(DPCServicePushMetricJob.java:69)
at com.sap.smd.server.exec.TaskRunner.run(TaskRunner.java:47)
at com.sap.smd.server.util.concurrent.PooledExecutor$Worker.run(PooledExecutor.java:785)
at java.lang.Thread.run(Thread.java:743)
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.GeneratedMethodAccessor5774.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at com.sap.smd.api.util.SynchronizedProxy$SyncHandler.invoke(SynchronizedProxy.java:27)
... 6 more
Caused by: com.sun.xml.internal.ws.client.ClientTransportException: HTTP transport error: javax.net.ssl.SSLHandshakeException: Received fatal alert: handshake_failure


JMS Scheduler - Missing Permissions for user J2EE_GUEST

The following exception occurs when the J2EE_GUEST user has no permission to perform the needed actions using the JMS scheduler queue.

javax.jms.JMSSecurityException: User: J2EE_GUEST has not permission: vpName: default, type: queue, action: browse, destination: SchedulerQueue
 at com.sap.jms.server.sc.UMESecurityProvider.checkPermission(UMESecurityProvider.java:223)
 at com.sap.jms.server.sc.UMESecurityProvider.checkDestinationBrowsePermission(UMESecurityProvider.java:126)
 at com.sap.jms.server.JMSVirtualProviderProcessor.consumerCreate(JMSVirtualProviderProcessor.java:352)
 at com.sap.jms.client.session.JMSSession.createBrowser(JMSSession.java:368)
 at com.sap.jms.client.session.JMSSession.createBrowser(JMSSession.java:341)
 at com.sap.sup.admin.scheduler.JmsScheduler.checkAndSendTasksToJms(JmsScheduler.java:1175)
 at com.sap.sup.admin.scheduler.JmsScheduler.access$000(JmsScheduler.java:84)
 at com.sap.sup.admin.scheduler.JmsScheduler$1.run(JmsScheduler.java:976)
 at com.sap.engine.core.thread.impl3.ActionObject.run(ActionObject.java:37)
 at java.security.AccessController.doPrivileged(Native Method)
 at com.sap.engine.core.thread.impl3.SingleThread.execute(SingleThread.java:185)
 at com.sap.engine.core.thread.impl3.SingleThread.run(SingleThread.java:302)

Recommended Solution:

Update your LM-SERVICE to SP 6 or higher.

Alternative Solution:

Ignore the error mes.sage as it does no crucial functionality is impaired.

Alternative Solution (Not Recommended):

Add the required permissions to the J2EE_GUEST user:

  1. Create a new role with a unique name possibly "JMS".

  2. In the assigned user choose J2EE_GUEST.
  3. In the assigned actions tab search for*jms*
  4. Choose "Service / Application: jms.default", "Name": "queue.all.all".
  5. Save the changes made.

Agent Authentication and Connectivity - Appendix

Connection Types

The Diagnostics Agent can connect to the Solution Manager via various protocols. The most commonly used combinations can be examine and modified via the Agent Administration's "Agent Connectivity" tab. If you need to change the connection settings of an agent that is currently not connected to the Solution Manager AS Java, you cannot use this UI. Instead, use the smdsetup script or manually edit the agent's runtime.properties as shown in the table below. The connection settings shown in the Agent Connectivity tab match the connection URLs specified in the runtime.properties. Use the table below with "myhost" being the host name of the Solution Manager AS Java, 1234 being the port number of the P4(SSL), and 5678 the port number of the Message Server HTTP(S) port.

Protocol
"smdsetup" Script Command
smd.agent.connection.url
Common Port Numbers
Direct P4 p4\://myhost\:12345<instance number>04
Direct P4 SSL p4s\://myhost\:12345<instance number>05
MS / P4managingconf hostname:"sapms://myhost" port:"5678"ms\://myhost\:5678/P48102
MS / P4 SSL ms\://myhost\:5678/P4S8102
MS-HTTPS / P4 mshttps\://myhost\:5678/P48103, 44402, or 44403
MS-HTTPS / P4 SSL mshttps\://myhost\:5678/P4S8103, 44402, or 44403

Using the Connection Log

To investigate connectivity issues, it is very helpful to use the Connection log of the Agent Administration. To enable it, open the Connection Log of the Agent Administration end adjust the settings as shown below.


P4 Logging on the Agent Side

  1. Stop the Diagnostics Agent (e.g. by executing  stopsap as  <SID>adm).
  2. Add the following snippet to the XML element log-destinations in /usr/sap/<SID>/SMDA<instance nr>/SMDAgent/configuration/log-configuration.xml.

    <log-destination count="10" effective-severity="All" limit="1000000" name="p4monitor" pattern="./log/p4monitoring.log" type="FileLog">
      <formatter-ref name="trc"/>
    </log-destination>
    <log-destination count="10" effective-severity="All" limit="1000000" name="p4" pattern="./log/p4.log" type="FileLog">
      <formatter-ref name="trc"/>
    </log-destination>
  3. Additionally, add the snippet below to the XML element log-controllers and save the file:

    <log-controller effective-severity="All" name="com.sap.engine.services.rmi_p4">
      <associated-destinations>
         <destination-ref association-type="LOG" name="SMDAGENT_ALL_TRACE_DESTINATION"/>
         <destination-ref association-type="LOG" name="p4"/>
      </associated-destinations>
    </log-controller>
    <log-controller effective-severity="All" name="com.sap.smd.agent.rmi_p4.monitor">
      <associated-destinations>
        <destination-ref association-type="LOG" name="p4monitor"/>
        <destination-ref association-type="LOG" name="SMDAGENT_ALL_TRACE_DESTINATION"/>
      </associated-destinations>
    </log-controller> 
  4. Start the Diagnostics Agent (e.g. by executing startsap as <SID>adm).

  5. All P4-related logs will be written to /usr/sap/<SID>/SMDA<instance nr>/SMDAgent/log/p4.*.log  or rather p4monitoring.log.

JSSE SSL Logging on the Agent Side

To enable SSL debug logging for connections that are established by the Java's default SSL engine JSSE, do the following. Although JSSE is used for most of the SSL connections, there are collectors (e.g. the SCC Collector (SAP Cloud Connector Collector) and the SAPPingHTTPCollector) that are instead using the IAIK library. The following configuration will only generate logs for SSL connections that are established by the JSSE engine. There is no way to get debug logs for connections established by the IAIK library.

  1. Stop the Diagnostics Agent (e.g. by executing  stopsap as  <SID>adm).
  2. Add the Java parameter -Djavax.net.debug=all to smdagent.javaParameters in /usr/sap/<SID>/SMDA<instance nr>/SMDAgent/smdagent.properties.
    If 
    usr/sap/<SID>/SMDA<instance nr>/smdagentgroup.properties exists (if Agents-on-the-Fly is enabled), you need to add the parameter in this file.

  3. Remove /usr/sap/<SID>/SMDA<instance nr>/profile/smd.properties.vmprop if it exists.

  4. Start the Diagnostics Agent (e.g. by executing startsap as <SID>adm).

  5. All SSL-related logs will be written to /usr/sap/<SID>/SMDA<instance nr>/work/jvm_smdagent.out.

Enable Server-Side Debug Logging 

In the NetWeaver Administrator go to Troubleshooting → Logs and Traces → Log Configuration. Switch to the Tracing Locations view and enter a package (e.g. com.sap.smd.*) or a class name (e.g. com.sap.smd.SMDManager), click on "Go" and select the desired location. Change the severity to Debug and click on "Copy to Subtree" in order to make the configuration effective for all class in packages. Afterwards click on "Save Configuration".

Mind that locations which have never logged anything before are not visible and cannot be selected. If you want to change the logging configuration for those locations, change the configuration for the closest parent and wait for its first log entry. You can now reset the configuration for the parent and change it for the dedicated location.

The logging configuration of a dedicated location can be reset to default by selecting the location and clicking on "Reset Location". Do not forget to also click on "Copy to Subtree" (if applicable) before saving the configuration. 

For general connectivity issues it is recommended to set the following locations to Debug:

  • com.sap.smd.server
  • com.sap.smd.SMDManager
  • com.sap.smd.SMDServerHandle

For P4-related issues, additionally increase the log level of:

  • com.sap.engine.services.rmi_p4
  • com.sap.engine.services.cross

Enable and Pull the NetWeaver Security Logs

  1. Open the Netweaver Administrator via  https://<j2ee_host>:<port>/nwa .
  2. Go to Troubleshooting → Logs and Traces → Security Troubleshooting Wizard.
  3. Click on Start Diagnostics
  4. Re-execute the operation you want to examine. E.g., if you want to examine the registration procedure of the Diagnostics Agent, re-start the agent and wait 1-3 minutes.
  5. Afterwards, click on Stop Diagnostics.
  6. Click on Download Zip Archive or rather directly view the logs in your browser.
     

ICM Logs of the Java Server 

The ICM logs can be found at  /usr/sap/<SID>/J<instance number>/work/dev_icm.

To increase the log level go to /usr/sap/<SID>/J<instance number>/exe and execute "icmon pf=/usr/sap/<SID>/J<instance number>/SYS/profile/<SID>_J<instance number>_<HOST>". The icmon program is interactive, this is to increase the trace level enter "+" and press enter. To quit enter "q" followed by enter. For further details see SAP Note 1095475.

ICM Logs of the ABAP Server (SMICM)

Open the SAP Gui transaction SMICM. Click on Goto → Trace File → Display All or rather Save Locally to view or rather save the logs. Click on Goto → Trace Level → Set/Increase/Decrease or rather Default to change the trace level or rather reset it. 

Solution Manager Server Thread Dumps

  1. Open a browser and access NetWeaver Administrator at http://<AS Java Hostname>:<HTTP Port>/nwa
  2. Navigate to Troubleshooting → Advanced Troubleshooting → Thread Dump Analysis.
  3. Click on "Generate Thread Dump", select "All Server Processes", click "OK".
  4. Download the generated thread dump.

Solution Manager Server Heap Dumps

In case of an out-of-memory error, the heap dumps of the Solution Manager are stored as  /usr/sap/<SID>/J<instance number>/j2ee/cluster/server<node number>/*.hprof.

Profiling the Solution Manager AS Java

In oder to analyse resource consumption issues of the Solution Manager AS Java (or any other SAP JVM process), trigger an automatic thread dump generation and import it to the SAP JVM Profiler or send it to the SAP support if requested. Furthermore, the SAP JVM Profiler can be used to trigger a Performance Hotspot Analysis.

Debugging the Solution Manager AS Java

To enable the debug mode of an AS Java, open the NetWeaver Administrator and go to Operations → Systems → Start & Stop → Java Instances. Select an instance an click on "Enable Debug" in the Java Processes tab. The port number will be displayed.


Using the Java Profiler

In rare cases you might be asked by the SAP support to use the Java Profiler in order to analyse very special performance issue within the Solution Manager AS Java. If so, do the following: 

  1. Install the Java Profiler
  2. On the Solution Manager AS Java host, start the daemon on the remote host by executing "./jvmmond" at sapjvm/bin/ of the AS Java.
  3. Prepare to reproduce the issue you have been asked to reproduce by the SAP support.
  4. Execute the Performance Hotspot Analysis
  5. Reproduce the issue.
  6. Click on "Stop analysis" once the issue has been covered.
  7. Export the result as an *.snp file and send it to the SAP support.
Method Parameter Analysis of Agent Connectivity-Related Issues
  1. Execute the “Method Parameter Analysis”

  2. Import the connectivity-related *.spf file as shown below:

  3. Start the analysis and reproduce the issue.
  4. Stop the analysis once after the issue has been reproduced and attach the exported analysis *.snp file.

Supported Web Dispatcher Versions

See SAP Note  2248724.

 

 

 

Troubleshooting Ressource Consumption and Booting Issues of the Diagnostics Agent

Check OS Settings on Linux Hosts (ulimit and umask)

After the installation of the Diagnostics Agent on UNIX platforms, remind to double check the following OS environment settings for the Diagnostics Agent OS user (often daaadm):

ulimit

  • Log on with the agent OS user, e.g. su – daaadm
  • Open an sh shell with the command sh
  • Execute the command ulimit -a
  • Compare the values with the recommendedations in the Installation & Setup Guide for the most recent installer (see SAP Note 1833501).

  • If you change the limits, remember to stop the Diagnostics Agent (sapstop), kill the Diagnostics Agent processes (sapstartsrv) and start the Agent again (sapstart). In this way the Diagnostics Agent will take the new settings into account.

If the user’s limit parameters for the OS Diagnostics Agent user are not correct, high CPU consumption situations or out-of-memory errors can result.

umask

Please refer to SAP note 1163751.

Trigger Thread Dumps for Diagnostics Agents

It is important to do thread dumps of the Diagnostics Agent process, to understand the problem context in case of a deadlock, not responding Diagnostic Agent or a high CPU time consumption situation.

Trigger a Thread Dump on Windows

The Diagnostics Agent is an SAP system and available in SAP MMC. 

                          
In SAP MMC, open the node “AS Java Process Table” of the Diagnostics Agent identified by the SID and instance number. Select the desired Agent Node and right click on it to display the context menu, then click the item “Dump Stack Trace”.

 

Trigger a Thread Dump on Unix

For Diagnostics Agents 7.1x, 7.2x, 7.3x or installed with SWPM 1.0 SP 3 and higher, log on in a UNIX shell with the OS user of the Diagnostics Agent, navigate to the folder /usr/sap/<SID>/SMDA<instance number>/exe and execute command:

sapcontrol.exe -user <SID>adm <PWD> -nr <Instance number> -function J2EEControlProcess smdagent DumpStackTrace

example:

./sapcontrol -user daaadm abc123 -nr 98 -function J2EEControlProcess smdagent DumpStackTrace

Alternatively a thread dump can be triggered via "kill -3 <pid>", with <pid> being the process ID of the Diagnostics Agent's Java process.

Where Is the Thread Dump Stored?

The  thread dump is written into the file /usr/sap/<SID>/SMDA<instance number>/work/std_SMDAgent.out, provided that the agent is running on  a SAP JVM.

Trigger Heap Dumps for Diagnostics Agents

See Using JVMMON to Trigger a Heap Dump

Verbose Class Loading

To enable verbose logging of the class loading of Diagnostics Agent add -verbose:class-verbose:class to the property smdagent.javaParameters in /usr/sap/<<SID>>/SMDA<<INSTANCE_NUMBER>>/SMDAgent/smdagent.properties. Restart the Diagnostics Agent and find the verbose class loading information at usr/sap/<<SID>>/SMDA<<INSTANCE_NUMBER>>/work/jvm_smdagent.out.

Common Issues Related to Ressource Consumption and Booting of the Diagnostics Agent

The Diagnostics Agent Log Contains Out-of-Memory Errors caused by abapReadSyslog

Symptom:

The log contains an exception like this:

[MAIJobObserver] ERROR occurred for metric collection 00000000000000000003[com.sap.smd.mai.model.collector.SAPControlWSCollector].
[EXCEPTION]
com.sap.smd.agent.plugin.connectors.webservice.WebServiceInvocationException: Webservice invocation error occured on BindingProvider JAX-WS RI 2.1.6 in JDK 6: Stub for http://mySystem:myPort/SAPControl.cgi
        at com.sap.smd.agent.plugin.connectors.webservice.JaxWebserviceInvocationHandler.invoke(JaxWebserviceInvocationHandler.java:114)
        at com.sun.proxy.$Proxy36.abapReadSyslog(Unknown Source)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at com.sap.smd.agent.facade.hostagent.HostAgentSyncProxy$SyncHandler.singleInvoke(HostAgentSyncProxy.java:127)
        at com.sap.smd.agent.facade.hostagent.HostAgentSyncProxy$SyncHandler.invoke(HostAgentSyncProxy.java:85)
        at com.sun.proxy.$Proxy43.abapReadSyslog(Unknown Source)
        at com.sap.smd.mai.model.collector.SAPControlWSCollector.abapReadSysLog(SAPControlWSCollector.java:1112)
        at com.sap.smd.mai.model.collector.SAPControlWSCollector.collect(SAPControlWSCollector.java:217)
        at com.sap.smd.mai.job.MetricJobRunner.run(MetricJobRunner.java:32)
        at com.sap.smd.server.exec.TaskRunner.run(TaskRunner.java:47)
        at com.sap.smd.server.util.concurrent.PooledExecutor$Worker.run(PooledExecutor.java:785)
        at java.lang.Thread.run(Thread.java:763)
Caused by: java.lang.OutOfMemoryError: GC overhead limit exceeded

Cause:

The syslog size is at least nearly as big as the Java maximum heap size.

Resolution:

  • To resolve the issue increase the Java maximum heap size (see below) to be sufficiently bigger than the syslog size.
  • Alternatively, limit the size of the syslog or delete it.

 

The Diagnostics Agent Log Contains other Out-of-Memory Errors

 

Symptom:

java.lang.OutOfMemoryError: Java heap space and java.lang.OutOfMemoryError: GC overhead limit exceeded can be found in the log file.

Cause:

By default, the SMD Agent's Java heap maximum size is set to 256 MB.  In the event of a large data collection, a heap size of 512 MB or more is required.

Resolution:

To resolve the issue, do the following for the Diagnostic Agent on host :

1.  On your managing host, browse to the following directory:

     Windows      <DRIVE>:\usr\sap\DAA\SMDA98\SMDAgent

     Unix      /usr/sap/DAA/SMDA98/SMDAgent

     Note: In the above example DAA is the name of the SID and the instance number is 98.  This may differ on your system.

2.  Open the file smdagent.properties using a text editor

3.  Locate the property smdagent.javaParameters then locate the parameter -Xmx

4.  Change the value of -Xmx to -Xmx512m then save the changes.  For example:

smdagent.javaParameters=-DP4ClassLoad=P4Connection -Xmx512m -Xms256m -XX:MaxPermSize=128m

5.  Restart the managing SMD Agent.

The Diagnostics Agent Does Not Start

For issues related to the start-up of the Diagnostics Agent, please refer to this Wiki page.

Extracting Configuration From Secure Store File Failed

If the SMDSystem.*.log contains errors similar to the entries listed below, the secure store of the Diagnostics Agent might be broken due to incompatible updates of the Diagnostics Agent or the JVM:

java.io.IOException: javax.crypto.BadPaddingException: Invalid PKCS#5 padding length: 33
at javax.crypto.CipherInputStream.a(DashoA13*..)
at javax.crypto.CipherInputStream.read(DashoA13*..)
at javax.crypto.CipherInputStream.read(DashoA13*..)
at java.util.Properties$LineReader.readLine(Properties.java:434)
at java.util.Properties.load0(Properties.java:353)
at java.util.Properties.load(Properties.java:341)
Extracting configuration from secstore file /usr/sap/DAA/SMDA98/SMDAgent/applications.config/com.sap.smd.agent.application.e2emai.std.collectors/_Default_Configuration.properties Failed. 
The secret key could not be read in secure storage 
com.sap.security.core.server.secstorefs.FileInvalidException: Getting Secure Store failed: File "/usr/sap/DAA/SMDA98/SMDAgent/configuration/secstore.properties" is invalid: software version 6.30.000.001 is incompatible with file version 7.00.000.001.
File "/usr/sap/DAA/SMDA98/SMDAgent/./temp/smdserver/secstore/TechnicalSecStorea8e9da9b2f1d7a8a9b079d5d1804a6c4.properties" is invalid: record with key "$internal/mode" is missing
Unable to create SSLContext because of KeyStore Exception java.security.UnrecoverableKeyException: Cannot recover key


To reset the secure store, proceed as follows:

  1. Stop the Diagnostics Agent
  2. Delete  /usr/sap/<SID>/SMDA<instance number>/SMDAgent/configuration/secstore.properties.
  3. Delete /usr/sap/<SID>/SMDA<instance number>/SMDAgent/configuration/security/.CertificatesKeyStore 
  4. Start the Diagnostics Agent.
  5. In the Agent AdministrationNon-Authenticated Agents select the respective Diagnostics Agent and click on Trust Agent.

In case the issue persists after the execution of the steps above, it is possible to follow the steps in the following article to correct this scenario.

  • 2447919 - Diagnostic Agent does connect to Solution Manager with the error: 'record with key "$internal/mode" is missing.'

The Diagnostics Agent is Blocking HANA Client Ports

When the Diagnostics Agent is started, all connections from the Solution Manager to the SAP HANA fail (e.g. Telnet on port 22 or the HANA port that is used by the Solution Manager server). 

Solution:

A possible cause might be that there are enough ports, but they cannot be reused fast enough. This can be configured using the below mentioned parameters. At SAP Note 2382421 especially check the sections "net.ipv4.ip_local_port_range" and "SAP Host Agent Configuration". The latter points to SAP Note 401162, which should also be taken into account. Furthermore, check the parameters "net.ipv4.tcp_tw_reuse" and "net.ipv4.tcp_tw_recycle" as described at SAP Note 2382421.

 

 

 

 

Agent Administration - Troubleshooting

Access to the "Diagnostics Agent Support Tool" and the "Agent-on-the-fly Administration"

If the links to the "Diagnostics Agent Support Tool" and the "Agent-on-the-fly Administration" are disabled, add the role SAP_RCA_AGT_ADM to your user. Use the transaction SU01.

 

 

 

Unable to render {include} The included page could not be found.

  • No labels