Registration

Dear SAP Community Member,
In order to fully benefit from what the SAP Community has to offer, please register at:
http://scn.sap.com
Thank you,
The SAP Community team.
Skip to end of metadata
Go to start of metadata

General Problem Handling

Please read the trouble shooting guide carefully. It contains several information to solve issues you might face with the Simple Diagnostics Agent. 

As a first action for problem detection check the Agent Errors in the Agent Admin UI. 

For any installation/configuration issues, check the installation and configuration logs in the Agent Admin UI. For any operational issue, check the Agent Errors in the Agent Admin UI. If you are not able to solve your problem by your own, open a ticket at SV-FRN-INF-SDA and upload the log files of the agent (/usr/sap/hostctrl/SMDAgent/default/logs and /usr/sap/hostctrl/work/sapstartsrv.log). You can use the Download Log Files from the Agent Action menu in the Agent Admin UI. In case this does not work, pick up the files form host.

Installation Issues

Unknown Operation DownloadSimpleDA

Issue: After setting up the SHA, adding the DownloadSimpleDA operation and triggering the Outside Discovery the SDA Installation fails with errors like "Unknown Operation DownloadSimpleDA" Solution: Check the access rights and ownerships of the operation and its directory to look like that:

 

# ls -la /usr/sap/hostctrl/exe/operations.d total 24
drwxr-x--- 2 root sapsys 4096 May 24 16:30 .
drwxr-x--- 7 root sapsys 4096 May 24 16:30 ..
-rw------- 1 root root   1286 May 24 15:06 DownloadSimpleDA.conf

 

If it is not like this, change it a accordingly and restart the Host Agent:

/usr/sap/hostctrl/exe/saphostexec -restart

Failed to install agent: "Failed to start /usr/sap/..../SimpleDA.lmsldesc" (Solution I)

Issue: File /usr/sap/hostctrl/exe/descriptors.d/SimpleDA.lmsldesc is ignored by the SHA because permission rules are violated.

Solution:

  1. Remove the file /usr/sap/hostctrl/exe/descriptors.d/SimpleDA.lmsldesc
  2. Stop any SDA related OS processes.
  3. Start the installation procedure again.

Failed to install agent: "Failed to start /usr/sap/..../SimpleDA.lmsldesc" (Solution II)

Issue: When trigger an installation of the Simple Diagnostics Agent, either automatically or by a manual execution in the Agent Admin UI, you get a similar error like

Failed to install agent: Internal Server Error Failed to deploy SDA: Failed to start /usr/sap/hostctrl/work/tmp_2019_05_20_13_05_27_284429_4100640496/agent/descriptors.d/SimpleDA.lmsldesc.

Related to the installation operation, in the sapstartsrv.log of the SAP Host Agent (/usr/sap/hostctrl/work on linux or C:\Program Files\SAP\hostctrl\work on windows) you see the error

[Thr 140434908546816] *** ERROR => Could not open directory '/usr/sap/hostctrl/exe/operations.d'

(13: Permission denied) [OperationDes 172]
[Thr 140434908546816] *** ERROR => SLPProxyHandlerFactory::ConnectContext called but ctx->operationID is NULL [SLPProxyHand 514]
[Thr 140434908546816] *** ERROR => Unable to open connection file $[DIR_HOME:#sapparam]/../SMDAgent/$[SDAName:#required]/agent/port [resolved as /usr/sap/hostctrl/work/../SMDAgent/default/agent/port]
(2: No such file or directory) [SLPProxyHand 713]

Reason: The directory  /usr/sap/hostctrl/exe/operations.d does not have sufficient file permissions.

Solution: Change the file permissions accordingly by execute  

/usr/sap/hostctrl/exe # chown root:sapsys operations.d 


Failed to install agent: "
Failed to start /usr/sap/..../SimpleDA.lmsldesc" (Solution III)

If none of the solutions provided above does not help, proceed as follow:

  1. Increase the trace level on SAP Host Agent as described here 
  2. Add rdisp/TRACE_LOGGING = on,100 m into the host profile
  3. Restart SAP Host Agent
  4. Trigger the SDA deployment (please explain how you do this)
  5. Download the sapstartsrv.log from /usr/sap/hostctrl/work
  6. Create a ticket at component SV-FRN-INF-SDA and attach the sapstartsrv.log file.

Missing Master Password File

Issuehttp 512: HTTP error 512; The following error occurred in the Simple Diagnostics Agent: com.sap.solman.agent.core.conf.ConfigurationException org.eclipse.equinox.security.storage.StorageException: No password provided. 

Reason: File /usr/sap/hostctrl/SMDAgent/default/configuration/agent/#/master_password does not exists. It seems that there are already configurations on SDA available but cannot be read as the password file is not in place. Typical issue is the copy from one SDA to another.

Solution:

  1. Delete the folder /usr/sap/hostctrl/SMDAgent/default/configuration
  2. Start a new configuration from Agent Administration UI.

Illegal Permissions on Master Password File

Issue: Error message: http 500: Internal Server Error; The following error occurred in the Simple Diagnostics Agent: com.sap.solman.agent.core.error.SystemException illegal permissions on master password file

Reason: File /usr/sap/hostctrl/SMDAgent/default/configuration/agent/#/master_password has wrong permissions. It seems that the OWNER_READ bit is not set for the master password file.

Solution:

  1. Ensure that /usr/sap/hostctrl/SMDAgent/default/configuration/agent/#/master_password has the correct permissions, e.g. -r-------- 1 sapadm sapsys 256 Nov 12  2017 master_password
    On Windows, the sapadm user must have full control to this file. This must be the only entry in the ACL.
  2. Restart the agent using the Agent Action "ping" in the Agent Administration UI.

Configuration of the SDA HTTP Port: java.net.BindException: Address Already in Use

Symptom:  On the monitored host, the default Simple Diagnostics Agent HTTP inbound port '3005' might be occupied by another system process. If so, the Simple Diagnostics Agent cannot start and the SAP Host Agent returns the HTTP request with the response "HTTP 500".

Issue: The SDA cannot be started because the reserved tcp port '3005' is already bound by another process.

Solution: The tcp port '3005' is usually reserved for the http port of the SDA. So, first try to switch the process, that blocks the SDA to another port. If this is impossible, follow SAP note https://launchpad.support.sap.com/#/notes/2549424.

Cannot Execute Java VM

Issue: java.exe cannot be executed in the SDA installation directory (e.g. SMDAgent/default/sapjvm/bin/java -version).

Reason: The msvcr100.dll is missing on the operation system.

Solution:  Apply the SAP note 1837221 (SAP JVM requires the Microsoft Visual C++ 2010 runtime).

General SDA Checks

All these checks can be done once the SDA has been deployed.

Check SDA Connectivity

  • At the Agent Action select Ping Agent and press Go.
  • You should get a pop-up containing some JSON formatted information.
  • if there is no pop-up
    • have a look at the Info Area at the top of the screen for some error messages
    • if you suspect connectivity issues
      • check the connectivity in transaction SM59

Check SDA Installation

  • in the details pane have a look at the Installation Log Messages. If you like to update the SDA, select the host and choose update from the selection menu.
  • !!! Unfortunately the window won't be refreshed, you have to search again to get the current information displayed !!!
Quick checks (Unix, adapt accordingly for Windows)
clear ; \
echo '=== SHA installed? ===' ; ls -l /usr/sap/hostctrl/ ; cd /usr/sap/hostctrl/ ; \
echo '=== SLD Destination configured? ===' ; ls -l exe/slddest.cfg* exe/config.d/slddest* ; \
echo '=== SLDREG run? ===' ; ls -l work/outsidediscovery.log ; \
echo '=== SDA installed? Version? ===' ; grep version SMDAgent/default/agent/configuration/agent/default.properties ; \
echo '=== Java Symlink present? ===' ; ls -l SMDAgent/default/sapjvm ; \
echo '=== Java startable? Version? ===' ; SMDAgent/default/sapjvm/bin/java -version ; \
echo '=== SDA running? ===' ; ps -fu sapadm ; \
echo '=== SDA pid correct? ===' ; cat SMDAgent/default/agent/pid ; \
echo '=== SDA password file permissions correct? ===' ; ls -l /usr/sap/hostctrl/SMDAgent/default/configuration/agent/#/master_password ; \ 
echo '=== port file present? ===' ; cat SMDAgent/default/agent/port

Expected Output

=== SHA installed? ===
total 12
drwxr-x---  3 sapadm sapsys 4096 Mar  3 13:41 SMDAgent
drwxr-x--x  6 root   sapsys 4096 May  4 18:32 exe
drwxr-x--- 11 sapadm sapsys 4096 May  4 18:32 work
=== SLD Destination configured? ===
-rw-r----- 1 root root 114 Jan 24 15:57 exe/slddest.cfg
=== SLDREG run? ===
-rw------- 1 root root 1049136 May  4 18:08 work/outsidediscovery.log
=== SDA installed? Version? ===
version=1.10.0.201705041226
=== Java Symlink present? ===
lrwxrwxrwx 1 sapadm sapsys 47 May  4 16:39 SMDAgent/default/sapjvm -> /usr/sap/hostctrl/SMDAgent/default/sapjvm_8_jre
=== Java startable? Version? ===
java version "1.8.0_112"
Java(TM) SE Runtime Environment (build 8.1.028)
SAP Java Server VM (build 8.1.028 25.51-b02, Dec 14 2016 16:11:21 - 81_REL - optU - linux amd64 - 6 - bas2:282279 (mixed mode))
=== SDA running? ===
UID        PID  PPID  C STIME TTY          TIME CMD
sapadm   12082 29457  0 17:44 ?        00:00:00 /bin/sh ./go.sh
sapadm   12140 12082  2 17:44 ?        00:01:31 ../sapjvm/bin/java -server -XtraceFile=../log/vm-trace.log -XX:OnOutOfMemoryError=./oom.sh -Xmx512m -XX:+DisableExplicitGC -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=../heap-dump -XX:HeapDumpOnDemandPath=../heap-dump-OnDemand -XX:ThreadDumpPath=thread-dump -Declipse.ignoreApp=true -Djava.awt.headless=true -Dj
sapadm   22143     1  0 18:32 ?        00:00:01 /usr/sap/hostctrl/exe/sapstartsrv pf=/usr/sap/hostctrl/exe/host_profile -D
=== SDA pid correct? ===
12140
=== SDA password file permissions correct? ===
-r-------- 1 sapadm sapsys 256 Nov 12  2017 master_password
=== port file present? ===
loopback:3005

 

Check SDA Configuration

  • in the details pane have a look at the Configuration Information
  • find the last configuration you are looking and check the content by clicking the link in the Status, Version column to see the content of a particular configuration message
  • click the link in the Error Message column to see the full text of it.
  • The Configuration status is:
    • White if the Diagnostics Agent is not configured. This should not be happen because there is always a default configuration at installation time. If you see a white configuration status anyway, do the following check
      • Open the 'LMDB' application
      • Search the host and open it by click on 'Display' after selecting the host in the result table
      • Get the timestamp of 'Last Change by Data Supplier'
      • Open SM37
      • Find Enter in the simple job selection
        • 'Job name': SAP_LMDB_NOTIFY_LDB*
        • 'User name': FRN_BTC_LDB
        • 'Job Start Condition': enter a time around the timestamp (-/+ 1h) you found in the LMDB
      • Execute the search
      • Check of all jobs were 'Finished'
      • Otherwise find the root cause in the job log and create a ticket for SV-FRN-INF-LDB
      • It is possible to execute the installation manually by using the 'Agent Administration'
    • Green means that the metrics are sent successfully from the Solution Manager to the Diagnostics Agent. The status can have two different  characteristic Comfirmed or up-to-date, if the status is Confirmed than the agent has the configuration versions that it has expected. If the status is up-to-date, the confirmation by metrics is still missing. 
    • Yellow pending: the SRSM can’t send any metrics to the Diagnostics Agent but it is assumed that it will be successfully send later. For that reason the SRSM tries to send the configuration in an interval and the status is “pending”. This happens e.g. if network problems occur, the authorization fails or the agent is not available for the moment.
    • Yellow Confirmation outdated means that the SRSM has sent metrics to the Diagnostics Agent but the metric version is older than expected. The SRSM tries it later again (“selfhealing”). You can identify this in the configuration information, looking for the Send Count > 1.
    • Red if an error occurred on Diagnostics Agent side and it cannot manage the configuration. 

Check SDA State by Using the Agent Administration UI

  • Start the Agent Administration form FIORI launchpad.
  • Search the corresponding host.
    • If the host name is known, go to Host Search
    • otherwise, go to Technical System Search.
  • In the result list, select the corresponding row.
  • The Diagnostics Agent's state is
    • White if the self-monitoring is not yet configured (currently part of the MAI configuration which is in Managed System Setup/Simple System Integration)
    • Green if the heartbeat is received regularly at the SRSM
    • Red if the heartbeat is configured, but is not received

SDA Availability State 'GREEN'

Configuration Status "Confirmed"

Issue: no issue. The SDA is working perfectly.

Solution: If still something is missing, re-execute the Simple System Integration configuration.

Configuration Status "Confimation outdated"

Issue: no issue. The SDA is working perfectly.

Solution: Wait some time until the confirmation is calculated. This is only a timing issue.

Configuration Status "Up-to-date"

Issue: no issue. The SDA is working perfectly.

Solution: Wait some time until the confirmation is calculated. This is only a timing issue.

Configuration Status "Error"

Issue: The SDA is still able to send metrics but is not able to retrieve new configurations or updates. Typically the SDA is in an inconsistent state due to update issues but the OS process is still running. 

Solution: Click on "Error" and follow up with SDA Error Messages

Configuration Status "Pending"

Issue: The SDA is still able to send metrics but is not able to retrieve new configurations or updates. Typically the SHA (SAP Host Agent) is not available/reachable.

Solution: Execute the PING agent action for the specific host and follow up with SDA Error Messages    

SDA Availability State 'GREY'

Configuration Status "Pending"

Issue: The SDA did not receive its self monitoring configuration. Typically the SHA (SAP Host Agent) is not available/reachable.

Solution: Execute the Agent Action "PING" and follow up with SDA Error Messages regarding the ping result.

SDA  Availability State 'RED'

Configuration Status "Error"

Issue: The SDA is not able to send metrics and not able to retrieve new configurations nor updates. The SDA is in a bad shape.

Solution: Click on "Error", select the first line in Configuration tab with a red configuration status and follow up with SDA Error Messages regarding the displayed error.

Configuration Status "Pending"

Issue: The SDA is not able to send metrics and is not able to retrieve new configurations or updates.

Solution: Execute the Agent Action "PING" and follow up with SDA Error Messages regarding the ping result.    

Configuration Status "Confimation outdated"

Issue: The SDA is not able to send metrics but was able to retrieve new configurations or updates. The server is working on a configuration confirmation

Solution: Wait some time and press "Refresh".  This is only a timing issue.

Configuration Status "Up-to-date"

Issue: The SDA is not able to send metrics but was able to retrieve new configurations or updates. The server could not confirm the configuration so far.

Solution: Open the Agent Internal application from Agent Action drop down. Check in Application "agent", if the value for URL and e2e.mai.user are correct and on your FRUN server, if the e2e.mai.user is not locked. If the "agent" application is not configured correctly, re-configure the SDA by starting "Configure Agent" from Mass Action menu entry. If the user is locked, unlock the user. If the unlock still occurrs, check, if the SDA's that should connect against the FRUN system have the correct credentials.

SDA Error Messages

Bad credentials in SM59 destination xxx

Reason: The destination that is used to create a connection between FRUN and an SDA has wrong credentials for the sapadm user of SAP Host Agent.

Solution: Adjust the credentials in the corresponding destination on the FRUN system. In case you need to reset the credentials, consult the SAP Host Agent administrator.

Connect to <hostname>:1128 failed: NIECONN_REFUSED(-10)

Reason: On the host, there is no SAP Host Agent installed. 

Solution: The solution depends on the result that is expected. If the host is

  • still valid
    • start with the (re-)installation of a SAP Host Agent
  • invalid
    • start the cleanup of the host. The cleanup is a complex procedure and documented in Decommissioning

Connect to <hostname>:1128 failed: NIEHOST_UNKNOWN(-2)

Reason: The FRUN system is not able to create a connection from the FRUN system to the SDA. The network is not able to find the host. 

Solution: The solution depends on the result that is expected. If the host is

  • still valid
    • start with the (re-)installation of a SAP Host Agent
  • invalid
    • start the cleanup of the host. The cleanup is a complex procedure and documented in Decommissioning

Connection to partner broken

Reason:

Solution:

Connection to partner timed out after 180s

Reason: The SAP Host Agent is not able to connect to the SDA via HTTP, but the SDA process is still up and running. Usually the root cause is an issue with the SDA update process. 

Solution: Stop the SDA process on the host.

  • Windows
    • Start the Task Manager, find the java process and stop it.
  • Linux
    • execute 'ps -fu sapadm' to find the running process (<pid>). Stop the process by 'kill <pid>'.
  • In the Agent Administration select the host and execute the Agent Action 'PING'. This operation should restart the agent. It cannot be precluded, that this results aftereffects.

Error: HTTP 404 Not Found

Reason: The SAP Host Agent is not able to create a connection to the Simple DIagnostics Agent. In the directory of the SDA a port file is missing. Usually the file will be created, when the SAP Host Agent starts the SDA for the first time. But the start of the SDA might fail if on the host, several processes of the SDA were started.

Solution: Stop all java processes of 'sapadm' user, Stop all /bin/sh ./go.sh processes of user 'sapadm'. Re-Install the SDA by executing the Mass Action "Install/Update Agent" in the Agent Management Application (Agent Admin UI).

Error in SDA Configuration

The issue can be seen, if the Agent Action "Configure Agent" ended with "Error". In this case, click on the error message link in the configuration tab. If you see the error details like:

http 500: Internal Server Error; Internal Server Error
Operation [id=00505694209A1ED7A8D674965B8A62D9] terminated with errors : 127 () 
description:Start SDA
Error: could not open `C:\Program Files\SAP\hostctrl\SMDAgent\default\sapjvm\lib\amd64\jvm.cfg'
exitcode:1

 

Reason: The SDA is not installed or in a corrupt state.

Solution: Re-Install the SDA by using the Agent Action "Install/Upgrade Agent". Re-trigger the "Configure Agent" action afterwards. Ensure that no old SDA process is running (otherwise kill the process) and there is enough disc space left to install and operate the SDA before re-installation will be triggered. Refer SAP Note 2501820 - Simple Diagnostics Agents needed Disk Space . 

Unable to Redeploy an SDA (Install After Manual Deletion)

It is not possible to install/update an SDA after manual deletion of the SDA on the host. The installation procedure skips the installation with the notice "SDA and JRE are already up-to-date, skip updating binaries"


Reason: During the installation procedure it is checked, if the version, that will be installed is newer then the existing one. If they are identical, the installation will be skipped to avoid unnecessary load in the system. The existing version of the SDA was recognized in the LMDB with outside discovery data upload.

Solution: The version in the LMDB must be changed so that the comparison returns a difference in the existing and the update version. Please follow the process

  1. Start the LMDB by selecting the host in the Agent Admin UI and press "Display Host" button.
  2. Change into the "Edit" mode.
  3. Press the "Details" button
  4. Select the tab "Associated Instances"
  5. Click on "Diagnostics Agent default on ...." in the column "Associated Instance"
  6. Select tab "Properties", press "Edit" (confirm the pop up) and scroll to the Version attribute
  7. Change the value to something different than the current value and press "Read Only"
  8. Close the pop up and press "Save"
  9. Restart the update of the agent in the Agent Admin UI 

SLDREGStatus: ERROR

Symptom: The Mass Action Logs tab shows the message "Outside discovery executed with error: SLDREGStatus: ERROR".

Solution: The Outside Discovery configuration of the host has to be revised. Details about the cause of the error can be found at /usr/sap/hostctrl/work/outsidediscovery.log, e.g. via the File System Browser.

Incident Component: BC-CCM-HAG

DPCConsumer Could  Not Push Metric Data

Reason 1: The Simple Diagnostics Agent uses an outdated HTTP-client implementation.

Solution 1: Update the Simple Diagnostics Agent to version 1.19 or higher.

Reason 2: The Solution Manager or the network is under load and processing or transmitting is delayed.

Solution 2: Set the property servlet-time-out to 120 (default 60) (time in s) in the agent's configuration file (hostctrl/SMDAgent/default/configuration/agent/[SM]/custom.properties).

Incident Component: SV-FRN-INF-SDA

A Sync Point Is Missing in the Result from the ABAP System (Solution Manager XYZ). Check If the DPC Sync Point Functionality Is Available

Reason 1: The ST-PI version might be outdated. This can be checked at SAP Logon → System → Status ... → SAP System Data → Details.

Solution 1: To make the DPC_SYNCPOINT running, the ST-PI version must be at least 4.

Reason 2: Another reason could be a problem with time zone settings of the system. Check the TZCUSTHELP report in the statistic records in STAD. If it reads "System time zone or UTC timer is incorrect"

Solution 2: Correct the time zone settings.

Incident Component: SV-FRN-INF-PRV oder SV-FRN-INF-SDA

Additional helpful information for support:

  • Access right issue: Transaction ST01 -> Authorization Trace for user SDAGENT (or the one that is configured)-> activate -> wait 5 minutes ->deactivate -> analyse -> attach result to incident

Time zone issue: Transaction SA38 in managed system -> TZCUSTHELP -> Execute -> enter current date and time ->attach result to incident

Web Service invocation error occurred on Binding Provider

Reason: The SDA does not have appropriate privileges to call WS-methods at sap control.

Solution: Add the following line to the profile DEFAULT.PFL:  service/admin_users = sapadm. This is mentioned in the FRUN Expert Portal.

Troubleshooting Ressource Consumption and Booting Issues of the SDA

Troubleshoot Out-of-Memory Issues

In case of out-of-memory issue the SAP support might need heap dumps to analyze the issue. Those can be generated as follows:

  • If possible, the SDA automatically creates a heap dump in case of an out-of-memory error. If you have OS access to the affected SDA, check if the file /usr/sap/hostctrl/SMDAgent/default/agent/default/heap-dump exists and if so, save it. Additionally, save /usr/sap/hostctrl/SMDAgent/default/agent/error if existing.
  • Alternatively, you can manually trigger and download a heap dump of the SDA using the Agent Action Get Heap Dump in the Agent Administration. Mind that this only works if the SDA is running and connected to the FRUN server. If the out-of-memory reproducively occurs after a dedicated amount of up-time, this functionality can be used to get a heap dump some time before the actual out-of-memory error happens.
  • If the Get Heap Dump function cannot be used because the SDA does not connect to the FRUN server, you can locally generate a heap dump by using the jvmmon tool:
    • Execute /usr/sap/hostctrl/SMDAgent/default/sapjvm/bin/jvmmon.
    • If there is more then one Java process running, enter the process ID of the SDA process (to be identified comparing the process path). Otherwise, the process is selected automatically
    • Enter dump heap.
    • Enter exit.
    • The heap dump files heap-dump-OnDemand.hprof and heap-dump-OnDemand.addons can be found at /usr/sap/hostctrl/SMDAgent/default/.

Troubleshooting SAP Host Agent

In some cases it is necessary to increase the log level of the SAP Host Agent without restarting the agent itself. If you are hosting the SAP Host Agent on a Linux machine, you are able to change the log level in the following way:

  • User-defined signal 1 (SIGUSR1, USR1) → decrease the tracelevel
  • User-defined signal 2 (SIGUSR2, USR2) → increase the tracelevel

Example:

kill -USR2 <pid> 

Where <pid> is the process id of the saphostctrl process.

Running an SDA and a Diagnostics Agent (Solution Manager 7.2) on the Same Host

It is possible to run an SDA and a Solution Manager Diagnostics Agent (SMDA) on the same host but for flawless operations the SMDA needs to be configured for Host Agent outside discovery which is also used for the FRUN scenario. Therefore follow SAP Note 'Switch Outside Discovery from Diagnostics Agent to SAP Host Agent' and use the local System Landscape Data Router (SLDR) for the SLD parameters in step 6.1. See also Preparing System Landscape Data Router for more details on the SLDR and its usage in mixed FRUN and Solution Manager operations.

 

Advanced Configuration

The SDA can be customised. All known options are documented in a dedicated page. Please check the Simple Diagnostics Agent - Advanced Configuration.

 

 

  • No labels