Changing the default trace level of SAP Host Agent

SAP Host Agent runs with tracelevel 1 by default. If you need more information to be able to analyze an issue, you can add or change the following profile values:

hostexec/trace
service/trace

Proceed as follows:

1. Open the SAP Host Agent profile ( host_profile ) which is located in the exe directory of the SAP Host Agent (/usr/sap/hostctrl/exe/ or C:\Program Files\SAP\hostctrl\exe\).
2. Add or modify the required values, for example:
   hostexec/trace = 3  
   service/trace = 3
3. Restart SAP Host Agent by executing saphostexec -restart (as a user with root authorization ) or hostexecstart -restart (as user <sapsid>adm)

To check the current tracelevel of the SAP Host Agent you can run the command below:

/usr/sap/hostctrl/exe/saphostctrl -function ExecuteOperation -name GetTraceLevel (as user root on Unix)
"C:\Program Files\SAP\hostctrl\exe"\saphostctrl -function ExecuteOperation -name GetTraceLevel (on Windows as user <sid>adm)

The output should contain the lines below if tracelevel=3 is activated:

----- Response data ----
sapstartsrv trace level: 3
saphostexec trace level: 3

All SAP Host Agent log files in /usr/sap/hostctrl/work/ (UNIX) or C:\Program Files\SAP\hostctrl\work\ (Windows) now contain much more information.

Dynamically changing the trace level of SAP Host Agent

To temporarly increase the trace level for performing some error analysis, please run the following command as root user on Unix:

/usr/sap/hostctrl/exe/saphostctrl -function ExecuteOperation -name SetTraceLevel level=3

or on Windows as user with administrative permissions, e.g. <sid>adm:

"C:\Program Files\SAP\hostctrl\exe\saphostctrl" -function ExecuteOperation -name SetTraceLevel level=3

After finishing the error analysis you can revert to the default trace level by running the command again with level=1.

Start of SAPHostExec Service / saphostexec executable failed. dev_saphostexec log file contains error message "ERROR => Setup of too many communication channels failed"

Reason:

The sapstartsrv process of the Host Agent is not able to start.

Solution(UNIX):

Proceed as follows:

1. Stop the Host Agent with saphostexec -stop .
2. Check that there are no running processes which belong to SAP Host Agent, and kill any remaining Host Agent's sapstartsrv processes.
3. Delete the /tmp/.sapstartsrv99_sapstartsrv.log and content of the Host Agent's work directory.
4. Restart SAP Host Agent by executing saphostexec -restart (as a user with root authorization) or hostexecstart -restart (as as user <sapsid>adm
5. Check the system log for the error messages from the sapstartsrv process of the Host Agent

Note:
Probably, there are multiple sapstartsrv processes running: One from the SAP Host Agent and one from each SAP instance. Just ensure that there is no SAP Host Agent sapstartsrv process running. You can identify it by the start parameter, which is pf=/usr/sap/hostctrl/exe/host_profile and by the assigned user, which is sapadm.

Other probable reasons for this error are:

1. /usr/sap/hostctrl/work and/or /usr/sap/hostctrl/work/sapstartsrv.* is not writable for the sapadm user
2. /tmp directory is not writable for the sapadm user or filesystem has no free space left.

The sapstartsrv error message in the system log should give you a hint why it cannot be started.

Solution(Windows):

Check the Application Log in the Windows Event Viewer for the error messages from the SAPHostControl service

Oracle databases cannot be detected on Unix

Symptom:

• Web service method ListDatabases does not return Oracle databases or not all Oracle databases
• Other database Web service methods (for example GetDatabaseStatus) return the error message "Database not found" for Oracle databases.
• With default trace level dev_sapdbctrl contains messages like
  [PID 23258] *** ERROR => Database neither found in '/var/opt/oracle/oratab' nor in inventory and environment of 'ora<sid>' user. Giving up... [dboractrlux.cpp 611]
• With trace level 3 (host_profile: service/trace = 3) the trace file /usr/sap/hostctrl/work/dev_sapdbctrl contains messages like the following:

[PID 10682408] Looking up databases in inventory
[PID 10682408] Retrieving inventory location from '/etc/oraInst.loc'
[PID 10682408] Retrieving inventory data from '/oracle/oraInventory/ContentsXML/inventory.xml'
[PID 10682408] *** ERROR => Failed to open file '/oracle/oraInventory/ContentsXML/inventory.xml': No such file or directory
[PID 10682408] No installations registered in inventory

or like the following:

[PID 10682408] Looking up databases in inventory
[PID 10682408] Retrieving inventory location from '/etc/oraInst.loc'
[PID 10682408] Retrieving inventory data from '/oracle/oraInventory/ContentsXML/inventory.xml'
[PID 10682408] No installations registered in inventory

Solution:

To correctly detect Non-RAC Oracle databases:

1. The database instance must be maintained in /etc/oratab (Solaris: /var/opt/oracle/oratab)
2. or a central Oracle inventory must exists and the ORACLE_HOMEs of all databases must be registered. Also the central inventory pointer file must exist and contain the correct path to the central inventory.

And for RAC databases:

1. SAP Host Agent 7.21 PL 14 must be installed
2. The GRID/ASM instance must be maintained in /etc/oratab (Solaris: /var/opt/oracle/oratab). E.g.
   +ASM1:/u01/app/12.1.0.2/grid:N
3. or a central Oracle inventory must exists and the ORACLE_HOMEs of at least the GRID/ASM instance must be registered. Also the central inventory pointer file must exist and contain the correct path to the central inventory.

Normally, all this is created by the Oracle/SAP installation tools.

For details on how to check the central inventory pointer file you can refer to e.g. http://docs.oracle.com/cd/E11882_01/em.112/e12255/oui2_manage_oracle_homes.htm#CJAEHIGJ. For details on how to check whether the central inventory exists and which ORACLE_HOMEs it contains you can refer to e.g. http://docs.oracle.com/cd/E11882_01/em.112/e12255/oui2_manage_oracle_homes.htm#CHDEHFBJ.

If the central inventory does not exist or it doesn't contain all ORACLE_HOMEs, you can create or adapt it with the Oracle runInstaller tool. For example, you can do this by executing the following command:

runInstaller -attachHome ORACLE_HOME="<Oracle_Home_Location>" ORACLE_HOME_NAME="<Oracle_Home_Name>"

For more information, refer to http://docs.oracle.com/cd/E11882_01/em.112/e12255/oui2_manage_oracle_homes.htm.

Wrong Oracle database status on Unix

Symptom:

Although the database is started and running fine it will be reported as stopped. E.g.:

# /usr/sap/hostctrl/exe/saphostctrl -function GetDatabaseStatus -dbname C11 -dbtype ora
Invalid databases status Database Status: Stopped
Component name: Instance (Instance), Status: Stopped (Instance is stopped)
Component name: Database (Database), Status: Stopped (Database is stopped)
Component name: Archiver (Archiver), Status: Stopped (Archiver is stopped)
Component name: Listener (Listener), Status: Running (Listener is started)

Solution:

1. Ensure that the correct Oracle home directory is either maintained in oratab or the inventory. Refer to Oracle databases cannot be detected on Unix for more details on that.

2. Ensure that the ora<dbsid> user exists and it has the correct environment (e.g. ORACLE_SID and ORACLE_HOME).
   
   The Oracle environment variables must be loaded via shell startup files (e.g. .cshrc in case of csh). You can use the following command to cross check the correct loading of the user environment (non-interactive login shell):

   su - ora<dbsid> -c env </dev/null | grep ORACLE_

    
   SWPM can be used to correctly setup/recreate the user.  You can refer to Note 2204211 for more details on this.
   Please note,  the ora<dbsid> user is always needed, also for 12c onwards and independently of the used user concept. You can refer to Note 1915323 for more details on that.

SAP Solution Manager Diagnostics (SMD) Agent is not able to connect to SAP Host Agent

We assume the <smd>adm user is used for the installation of the SMD agent installation, where <smd> is the system ID of the SMD agent ("DAA" by default).

Symptom:

The following issues may arise while the SMD agent is connecting to SAP Host Agent for the first time:

Trusted connect is not working

Solution:
Proceed as follows to check if SAP Host Agent is working correctly:

1. Log on on the respective server as <smd>adm.
2. Execute the following command (Note: Instance number is always 99. This is not an example!) :
   on UNIX:
   
   /usr/sap/hostctrl/exe/sapcontrol -nr 99 -user "" "" -function ConfigureLogFileList add /tmp
   
   on MS Windows (replace the <smdadm> and <password for smdadm> with your user and password):
   
   c:\usr\sap\DAA\SMDA97\exe\sapcontrol.exe -user <smdadm> <password for smdadm> -nr 97 -function OSExecute "\"c:\Program Files\SAP\hostctrl\exe\sapcontrol.exe \" -user \"\" \"\" -nr 99 -trace - -function ConfigureLogFileList remove testLogFile.txt" 0 0 c:\temp\tst_out.txt

If the output of the command prompt is as follows, the trusted connect works:

12.12.2012 11:25:27
RequestLogonFile
OK

12.12.2012 11:25:27
ConfigureLogFileList
OK

This generally means that Trusted Connect has been configured correctly on the part of SAP Host Agent.
However, the SMD Agent call might fail. In those cases you should proceed as follows:

Check if ACLs are used. In the host_profile file you can find a configured entry: service/http/acl_file=<filepath>
If yes, check in the ACL files if all the relative IP Addresses of the local machine are configured correctly. See SAP Note 2701115.

Alternatively you can proceed as follows:

1. In the host_profile, specify service/trace=3
2. Restart Host Agent with the following command: saphostexec -restart
3. Trigger the SMD scenario
4. Check if entries like NiAcl .... denied are contained in the sapstartsrv.log file. If yes, the ACL configuration is wrong and this is the root cause of that issue.

If the sapcontrol call does not return with OK, you have the following options to proceed:

The output is: "FAIL: Permission denied"

The output of the command prompt is as follows:

ConfigureLogFileList
FAIL: Permission denied

Solution:
Proceed as follows:

1. Check if the <smd>adm user is specified in the host_profile in line service/admin_users .
2. If <smd>adm user is not specified there, add it.
3. Restart the SAP Host Agent.

Remark: Also on Windows the service/admin_users in the host_profile are case sensitive and need to be defined exactly as the user running the service. For example service running as <DOMAIN_NAME>\SAPServiceDAA cannot be defined as <domain_name>\SAPServiceDAA in the host_profile. The upper and the lower case letters must match.

The output is: "HTTP/1.1 401 Unauthorized"

The output of the command prompt is as follows:

ConfigureLogFileList
FAIL: HTTP error, HTTP/1.1 401 Unauthorized

Analysis:

User <smd>adm seems not to be able to read the logon file. The cause might be that file system permissions or ownership are wrong.

Solution:

Ensure that the file permission and ownership are as follows:

# ls -lda /usr/sap/hostctrl/work/sapcontrol_logon
drwx--x--x 2 sapadm sapsys 4096 28. Apr 06:33 /usr/sap/hostctrl/work/sapcontrol_logon

Ensure that the <smd>adm user is member of the sapsys group and the user is able to access files in /usr/sap/hostctrl/work/sapcontrol_logon.

The output is: " OSExecute FAIL: Start Service runs with administrative privileges, OSExecute disabled"

The output of the command prompt is as follows:

OSExecute FAIL: Start Service runs with administrative privileges, OSExecute disabled

Analysis:

OSExecute is rejected because the User SAPService<SMD> has too many privilleges.

Solution:

Remove the user SAPService<SMD> from the group of Administrators and make sure to restart the <SMD> Service. Please note restarting the <SMD> service will restart the Diagnostics Agent as well.

Error message "Remote access not permitted" in SMD agent logs

Ensure that you have at least Host Agent 7.20 patch level 179 installed.

Autoupgrade from shared directory not working on Windows

For windows, the Autoupgrade has certain prerequisites:

1. The Host Agent system must have access to the share. The SAPHostExec service is doing the autoupgrade and this service is running as Local System Account. ( One can read more about the Local System Account on http://msdn.microsoft.com/en-us/library/windows/desktop/ms684190 ) Hence, the <System>$ account must have access to the shared directory.
2. Autoupgrade is working with UNC paths only. "Mounted" directories (e.g. as Drive X:) do not work.
   

If the Host Agent System and the System with the shared directory are located in different windows domains, there must be a one way trust to access the share. That means, the domain of system A (the system with the shared directory) has to trust the domain of system B (the system with the Host Agent).

See http://msdn.microsoft.com/en-us/library/cc237016%28v=prot.10%29.aspx for more information on how to setup a domain trust.

Unable to detect Microsoft IIS applications

Symptom:

The command to retrieve list of IIS web applications fails with error "Invalid namespace":

"c:\Program Files\SAP\hostctrl\exe\saphostctrl.exe" -function GetCIMObject -namespace WebAdministration -enuminstances Application -user sapadm <password>

Error: CIM Request EnumerateInstances for classes Application failed: EnumerateInstances for Application failed: Invalid namespace

Solution:

Please ensure that the "IIS Management Scripts and Tools" are installed. For example, with Windows 2008 this can be checked via Server Manager -> Roles -> Web Server -> Add Role Services.

Username/Password-based authentication fails on Unix

Symptom:

When a Web service method of the Host Agent is called or a Web-UI is accessed via the Host Agent (e.g. SUM) Username/Password-based authentication fails although the correct credentials were supplied.

Solution:

Ensure that the Host Agent is correctly installed/updated as described in Note 1031096. A common error is that the host agent files are manually copied into /usr/sap/hostctrl/exe or the host agent archive is just directly extracted into /usr/sap/hostctrl/exe. This results in invalid file ownership and permission, which in turn breaks various functionality like e.g. the Username/Password-based authentication.

To fix this install the Host Agent as described in Note 1031096.

Host Agent fails to load clean root user environment on startup or installation/upgrade

When starting or upgrading the Host Agent you see the following messages:

Executing "'/bin/su' '-' '-c' 'echo SAPHOSTAGENT-START;env;echo SAPHOSTAGENT-END' "
*** WARNING => Timeout, helper didn't answered within 30 sec...
*** ERROR => Unable to load clean root environement.

Starting with 7.21 PL8 the Host Agent is trying to load a clean root user environment during startup. This is done by running a (non-interactive) root user login shell. This step hangs and is therefore interrupted after some timeout.

Typical reasons for that are some prompts issued by shell login/startup scripts requesting manual user input or the script command for shell recording. You can check this by e.g. running the following command:

/bin/su - root -c env </dev/null 

Please ensure that these non-interactive root user login shells don't hang/require any user input. The shell login/startup scripts are shell-dependend and sometimes also OS dependent, e.g. on Linux a non-interactive login bash uses /etc/profile, ~/.profile (and sometimes ~/.bash_login), csh uses /etc/csh.cshrc, /etc/csh.login, ~/.cshrc and ~/.login.

Configuring supportability features for SAP Host Agent helper process (like sapdbctrl, sapcimc, sapcimb, lssap,...)

Crash Handler in case of abnormal termination 
Besides that a call stack is always written to the trace file, there are means to force the creation of a core / dump – independent from settings on the machine about resource usage limits (on UNIX platform) rsp. settings for Windows Error Reporting (WER):

Creation of a core / dump is forced by adding the name of the helper binary into a config file:

• • (UNIX)         /usr/sap/hostctrl/exe/config.d/core.conf
  • (Windows)  <SAP Host Agent exe dir>\config.d\minidump.conf

Example (UNIX):

#cat /usr/sap/hostctrl/exe/config.d/core.conf
sapdbctrl
sapcimb

For Windows Minidumps, additional optional settings are possible:

Related info:

Windows Error Reporting (WER)     https://msdn.microsoft.com/en-us/library/bb513638(VS.85).aspx
Collecting User-Mode Dumps   https://msdn.microsoft.com/en-us/library/bb787181(v=vs.85).aspx
1399014 - Windows Error Reporting as of Windows Server 2008

Example (Windows):

>type "C:\Program Files\SAP\hostctrl\exe\config.d\minidump.conf"
sapcimb
sapdbctrl
sapdsigner
MINIDUMP_TYPE   10
MINIDUMP_PATH   C:\tmp

sapgenpse troubleshooting

A common problem with sapgenpse in the context of SAP Host Agent is that the SECUDIR variable is not set correctly. sapgenpse does not require SECUDIR to work correctly as it will use default values if SECUDIR is not set. However, depending on the user, environment loaded, user switches et cetera this might be confusing (e.g. sapgenpse might assume different locations for PSE and the credentials file).

Setting the SECUDIR should be a safe option to prevent mistakes in this context.

To check SECUDIR location (and other values, too), you can use the support_info parameter of sapgenpse.

root@mymachine:/usr/sap/hostctrl/exe# su - sapadm -s /bin/bash -c "SECUDIR=/usr/sap/hostctrl/exe/sec sapgenpse support_info"
No directory, logging in with HOME=/
Version Information:
  Version:                  8.5.36
  Platform:                 linux-gcc-4.3-x86-64
CryptoKernel:
  Supported CPU features:   AES-NI,CLMUL,SSE3,SSSE3
  Active CPU features:      AES-NI,CLMUL,SSE3,SSSE3
  FIPS 140-2:               NO
Trace configurations (highest priority first):
  Profile:
    No configuration
  /sec/sectrace.ini:
    No configuration
  /usr/local/bin/sectrace.ini:
    No configuration
Environment:
  CCL_PROFILE:              <unset>
  SECUDIR:                  /usr/sap/hostctrl/exe/sec
Binaries:
  Binary directory:         /usr/local/bin
                               38320 sapgenpse
                             5486348 libsapcrypto.so
                              499679 libslcryptokernel.so
                                 166 libslcryptokernel.so.sha256
Configuration:
  Empty
PSE directory content:
  PSE directory: /usr/sap/hostctrl/exe/sec
    SAPSSLS.pse
  Credentials:
    /usr/sap/hostctrl/exe/sec/SAPSSLS.pse                            CN=192.168.0.1, O=SAP SE, C=DE

In the example above, SECUDIR was set explicitly when calling sapgenpse as shown in line 19 of the output. Note that the way it is called, the SECUDIR variable is only set in the context of this single sapgenpse command. When performing other sapgenpse commands you always need to pass the variable this way therefore. Instead, you may set SECUDIR in the user environment and make sure that it is available when invoking sapgenpse. You should check if the variable is set correctly via the support_info command again.

There are additional commands available to find out the root cause of an error like the "-log" and "-v" parameter of sapgenpse. 

Additional Information for sapgenpse

• Central Note for CommonCryptoLib 8 (SAPCRYPTOLIB)
• Configuring SSL for SAP Host Agent on UNIX

sapinit cannot be installed on SUSE Linux SLES 15/is not running

Running the commands below as user 'root':

cd /usr/sap/hostctrl/exe
./installsapinit.sh

returns an error "/sbin/insserv: No such file or directory"

The "insserv-compat" package is not installed. The "insserv-compat" package is part of the "sap_server" pattern. For more details refer to note # 2578899 - SUSE Linux Enterprise Server 15: Installation Note.

saphostctrl -https fails with "Error: SSL/TLS error, Server certificate mismatch"

Starting from SAP Host Agent 7.22 PL59 the command line tool saphostctrl with the option -https strictly verify the hostname of the SSL Certificate returned by the server.
If the hostname returned by the server certificate doesn't match exactely with the hostname used in the -host argument you may get the following error:

user@clienthost:~$ ./saphostctrl -https -host host1 -function <Method>
Error: SSL/TLS error, Server certificate mismatch
user@clienthost:~$

In such cases you have the following options:

1. Pass the proper hostname as argument to saphostctrl
   • e.g ./saphostctrl -https -host host1.mydomain.com if , for example, the Certificate Subject is "CN=host1.mydomain.com, O=MY Company, C=US"
2. Ignore the SSL certificate mismatch error using the argument -insecure
   • ./saphostctrl -https -host host1.mydomain.com -insecure
3. Configure SNI support for SAP Host Agent for each used hostname on the server host
   • SNI / multiple PSE Support