Page moved!
This page is outdated and it's content has been moved into the new structure under SAP HANA Managed System Setup in SAP Solution Manager.
This page will not be updated with findings or corrections.
Prerequisites
To be able to set up your SAP HANA with System Replication (SR) in the way described below please note that your SAP Solution Manager system needs to run on SAP Solution Manager 7.1 SP14 or higher. Only with this support package the necessary developments were shipped that enable the modeling of an SR SAP HANA database in SLD and LMDB.
If your SAP Solution Manager is running on SP 13 or lower the two (or) database systems of the SR scenario have to be set up as independent databases and reconfiguration will be necessary in case of a take-over to the SR site.
Apart from the SAP Solution Manager support package you have to make sure the necessary notes are implemented in SAP Solution Manager and the managed systems (ABAP and SAP HANA). You find all required notes here.
Your SAP HANA SR system must be set up following the recommendations in the SAP HANA installation guide and the primary system and the secondary system must be on two different hosts. Check in HANA Studio to see if the SAP HANA system is set up with SR and how this setup looks.
The exact setup of the system replication can be found under the "Landscape" tab.
Setup Steps of HANA DB and HANA Host
Install Diagnostics Agent and SAP Host Agent on HANA hosts
The agent installation follows the same principles as for standard SAP HANA installations. Please refer to the respective section in the wiki for more information.
Setup Monitoring User in HANA
The monitoring user in SAP HANA is created the same way as for the SAP HANA installation with SR. If you run the SAP HANA system with an embedded statistics server please make sure you add not only the MONITORING role but also the execution authorizations for the _SYS_STATISTICS schema as described in SAP note 1917938.
Please make also sure that you disable the password lifetime for the users to prevent it from getting locked because the password expires. This can be done using this SQL command. See also SAP note 1702224 for details.
Setup HDB User Store for SAP Host Agent
The setup of the HDB user store is performed the same way as for SAP HANA installation without SR. Please refer to Managed System Setup of HANA Standard Installations#SetupHDBUserStoreforSAPHostAgent for details. Please make sure you set up the HDB user store on the primary as well as on the secondary system.
Register HANA in SLD
One of the most important changes that enable us to model SAP HANA SR systems in SLD and LMDB is the implementation of a new association that enables us to model relationships between DB replication groups. This also lead to the introduction of a new parameter in the profile on the SAP HANA system. If all works correctly in the end the associations in the SLD should look like this:
In the graphic above you can see that each SLD DS creates an own CIM Object of Type SAP_Database_System.
The SAP_Database_System created by the SLD DS of the Application Server (1) is named “Virtual Database”, the SAP_Database_System(s) created by the HANA SLD DS (2) are named “Physical Database”. The relationship between the “Virtual Database” and the “Physical Database” which is the Primary DB is modeled with the Association SAP_IdenticalDatabaseSystem (3). The relationship between the Primary DB and the Secondary DB is modeled with the Associations SAP_DatabaseReplicationDependency (4).
To be able to report the system replication dependencies SAP_DatabaseReplicationDependency within SAP HANA you need at least SAP HANA revision 102.02 or higher. To be able to report SAP_IdenticalDatabaseSystem automatically via the parameter SLDVIRTDBHOME you need at least SAP HANA revision 110 or higher.
If your SAP HANA system doesn't meet the requirements you can model the dependencies manually in LMDB in SAP Solution Manager SP 14 or higher. Please still maintain all profile parameters as described in the following, to makes sure you have all technical systems reported to SLD and LMDB that you need to maintain the relations.
Maintain SLD parameters in SAP HANA and Application Server Profile
To be able to model the relationships above we had to introduce a new parameter for SAP HANA which will be handled as of revision 110. The parameter is called SLDVIRTDBHOME and contains the logical hostname under which the SAP HANA DB can always be reached. As the SAP HANA is installed as a SR system there must be one logical hostname that moves from the primary to the secondary site in case a fail-over to the replicated system happens. This is the host that has to go into the parameter SLDVIRTDBHOME.
Parameter SLDVIRTDBHOME
The profile parameter is only taken into account in SAP HANA system with rev. 110 or higher. You could still already maintain this parameter in the profile but for now it will be ignored. This parameter is needed to create the SLD association SAP_IdenticalDatabaseSystem between the current primary system and the virtual database system that is reported by the SAP application server. For SAP HANA systems with lower releases you have to create this relation in LMDB manually after the SLD registration. This is described later in this wiki post.
SLDSYSTEMHOME ≠ SLDVIRTDBHOME in HANA
Now even in a SR scenario you can still run your partaking SAP HANA systems as Scale-out installations. In a scale-out the SAP HANA system is installed with multiple hosts (multiple nodes). This has nothing to do with system replication, but still I have to mention it here, as this setup requires the already known parameter SLDSYSTEMHOME to be set up additionally to SLDVIRTDBHOME. Please find more details how and why to maintain the SLDSYSTEMHOME parameter for scale-out scenarios on the wiki page Managed System Setup of HANA Standard Installations#RegisterHANAinSLD.
Please note the SLDSYSTEMHOME on HANA should not be the same as SLDVIRTDBHOME on HANA in a SR scenario. For the SLDSYSTEMHOME parameter you choose one of the hosts that can run the indexserver in MASTER role for one database (which one is pretty much up to you, see the other wiki page above for details). However this hostname belongs to one specific SAP HANA database in your SR scenario. It cannot belong to both HANA databases in the SR scenario. And it would never move to another physical host! SLDVIRTDBHOME belongs to all HANA databases in your SR scenario and it moves to whichever the main host of the Primary is. Please refer to the example below for better understanding.
Also in the profile of the SAP ABAP Application server you might have to make a slight adjustment. The SAP ABAP system reports the database with the value SAPDBHOST. If SAPDBHOST and SLDVIRTDBHOME are the same, then there is no problem. During the SLD registration the following would happen:
- SAP ABAP reports a DB system with the parameter SAPDBHOST
- HANA reports that it is an SR scenario with a virtual DB host on SLDVIRTDBHOME
- In SLD the database formerly reported by HANA is changed to type "Virtual Database" and the relation SAP_IdenticalDatabaseSystem is created to the reporting HANA system
This naturally only works if SAPDBHOST on ABAP matches SLDVIRTDBHOME in HANA. If this is not the case, the mechanism would fail, and no virtual DB or relation SAP_IdenticalDatabaseSystem is created.
To address differences between SAPDBHOST and SLDVIRTDBHOME you can use the parameter SLDSYSTEMHOME on ABAP. So if SAPDBHOST on ABAP is not the same as SLDVIRTDBHOME on HANA, you can set the parameter SLDSYSTEMHOME on ABAP to the same value as SLDVIRTDBHOME on HANA.
In our example database we have exactly this case, we have a SAP HANA database that runs on two hosts with a MASTER and a SLAVE indexsever.
These two hosts are replicated to two other hosts with exactly the same setup:
The SAP HANA H40 is used by the SAP ABAP system HHA.
So to address this setup the following parameters were added to the SAP HANA DEFAULT.PFL of both SAP HANA systems.
The profile of the (current) primary system running on lddh1hha and lddb2hha:
The profile of the (current) secondary system running on lddb3hha and lddb4hha:
The logical hostname lddbhha is currently running on lddb1hha:
In the SAP ABAP system we added the parameter SLDSYSTEMHOME to the DEFAULT.PFL.
Now that we have prepared all the profiles, we can perform the registration of the system in SLD.
SLD Registration
The SLD registration is performed the same way as for a SAP HANA DB using HLM for SAP HANA with releases < SP08 or using HDBLCM for SAP HANA with releases >= SP 08. Both methods are described in the section Managed System Setup of HANA Standard Installations#RegisterHANAinSLD in the wiki for the standard SAP HANA managed system setup.
You must perform the configuration of the SLD data supplier on all SAP HANA systems partaking in the SR scenario using, e.g. HDBLCM. Also on the secondary system you can configure the SLD DS using HDBLCM on OS level, even if the system is in standby-mode. However please see the note below for current restrictions as to when the secondary system is actually registered in SLD.
SLD Registration of the Secondary System
Once the SLD registration is done the SAP HANA system will register itself in SLD on a 12 hour basis. However, there is one catch. The secondary system is in stand-by mode, and SAP HANA is a database system that runs on the "no read in standby" principle. This means it doesn't accept any SQL statements and certain processes are not started automatically. Currently one of these processes is the process starting the SLD DS, hence the SLD DS on the stand-by system (the secondary system) will not run until the first switch happens. So right now the only way to get the complete SAP HANA SR landscape in the SLD is indeed to perform one switch forth and back after the primary system is registered in SLD and the SLD data supplier is configured for the secondary system.
If your secondary SAP HANA system is already in SLD from a previous registration you don't have to worry about this.
If the secondary system is not in the SLD also the SR associations within SLD will not be created, as the associations cannot point into "nothing".
SAP HANA database Revision 110 - 122.04
With SAP HANA database Revision 110 - 122.01 there is a bug in the SLD data supplier that leads to an incorrect entry in the sldreg.xml. As an result the association SAP_IdenticalDatabaseSystem is not created. Please refer to SAP note 2303938 for a currently manual workaround.
Result
In summary you should see three HANA databases in the end. As before you have to go to the CIM Instances to see your HANA systems. In older SLD releases you have to use the path Administration => Content Maintenance to access the CIM Instances.
Filter for SAP_HDBSystem and the SID of your database. The three systems you see are:
- The HANA DB reported by the SAP ABAP system
- The HANA DB which is the current Primary System
- The HANA DB which is the current Secondary System
Unfortunately the registration of the systems in the SLD is not really working as I would want it to. Yet. We are working on getting this perfect, but this takes some more time. So I will describe here for each of the systems what you would want to see in the SLD and also what you might have to manually adjust, if you do not see what you want to see. In a later section I will describe what has to be adjusted in case of a take over (if you don't plan or cannot switch back to SITEA as primary).
SAP HANA DB reported by the SAP ABAP system
In this system you will not see much, as it is mainly an empty shell that serves as link between the ABAP system and the SR cluster. You recognize it by the hostname, which is usually the logical hostname that you maintained in the parameter DBSYSTEM of the ABAP stack. Important is the association SAP_IdenticalDatabaseSystem that should point to your current primary HANA system. Make sure you have this, because otherwise you don't have a link between the SAP ABAP system and the SR cluster.
SAP_IdenticalDatabaseSystem in SAP HANA < Rev. 110
As mentioned above this association is created by the parameter SLDVIRTDBHOME in SAP HANA rev. 110 or higher. Until then you should not see an association here. Please refer to the section of the LMDB maintenance to learn how to create this relation manually.
If you run SAP HANA between rev. 100 and rev. 110 and you do see this association please check the parameter sql_connect_hosts in the global.ini file. In this older revisions of SAP HANA the SLD DS wrongly used this parameter to create and adjust this association on a switch. If the parameter is set and it is not by chance the same hostname that you would maintain in SLDVIRTDBHOME then you will have to adjust the relation in LMDB manually.
SAP HANA DB which is the current Primary System
In the current primary SAP HANA system you want to see two associations that are related to system replication:
- SAP_IdenticalDatabaseSystem: Which points to the SAP HANA system reported by the ABAP system, you should only see this for SAP HANA rev. 110 and higher
- SAP_DatabaseReplicationDependency: in the Role Antecedent, which points to the current secondary system
Expert Step: Remove superfluous Association from SLD
If you made a full take-over and failback to create your secondary system in SLD, you will notice that there are two entries for SAP_DatabaseReplicationDependency. Please manually delete the one with the Role Dependent by selecting it and clicking on the "Remove" button on the current primary database system. This one is supposed to be removed automatically in future.
This will also automatically delete the corresponding faulty association in the SLD entry for the current secondary database system.
Manual Maintainance for SAP HANA < Rev. 102.02
If you have a SAP HANA system older than revision 102.02 the association SAP_DatabaseReplicationDependency is not reported to SLD. Please follow the manual workaround to model the relations in LMDB described below.
SAP HANA DB which is the current Secondary System
On your current secondary system you want to see the association SAP_DatabaseReplicationDependency in the Role Dependent.
Once you have these connections straight you can check the result in the LMDB of SAP Solution Manager.
Setup in Solution Manager and on Solution Manager Host
Install client software on SAP Solution Manager
The client software installation is independent of if you run an your SAP HANA with SR or not. Please refer to the section in the wiki for the standard SAP HANA setup for more information.
Set DB parameter on Solution Manager
Also here, please refer to the corresponding section in the standard SAP HANA setup wiki.
Check LMDB
Once the SAP HANA system and the ABAP stack are registered in SLD correctly they will be synchronized into LMDB. As in SLD you will see three SAP HANA system with the same SID. The extended SID for the systems will differ as the LMDB attached 0000N to the SID if two systems have the same one.
The extended SID are assigned depending on the order the systems are reported to LMDB for the first time. You could change them to something more meaningful. I decided to change them to H40SITEA, H40SITEB and just H40 for the ABAP reported database.
Please be careful when changing extended SIDs. You should only do this if the system is not used in any logical components, project or monitoring yet.
In general the relationships in the LMDB mirror the ones in the SLD. This is why it is important that we delete the superfluous association SAP_DatabaseReplicationDependency after a take-over. Otherwise LMDB would get confused. The relations are modeled like this:
So now lets have a look at the single databases in LMDB.
SAP HANA Database reported by the SAP ABAP System
This is the SAP HANA database system that is also referred to by the SAP ABAP system in LMDB.
When you open this database system you will see that it is maintained as Virtual Database.
With SP 14 we introduced the view "Related Databases" in LMDB. Click on this view and you will see that it is related to the current Primary database in the SR cluster. The current primary database is maintained in the role Physical database. This relation is created by the SLD association SAP_IdenticalDatabaseSystem.
SAP_IdenticalDatabaseSystem for SAP HANA < Rev. 110
If your SAP HANA system runs on a revision < 110 you will not see the relation to the current Primary database here. The reasons are explained above. In this case you have to create the relation manually in LMDB.
To create the relation click on "Assign Physical Database" and select the current Primary database of the SR scenario.
Save your changes!
Please note that you will have to adjust this manually created relation after the take-over. See the section "Reconfiguration Tasks after SR Take-Over" for details.
SAP HANA DB which is the current Primary System
For each database in LMDB we also maintain the role it has in an SR scenario (if it takes part in one). For the primary database you will see the role Primary Database. Also because it is an actual database it is maintained as Physical Database.
On the view Related Databases you see now the Virtual Database it is related to via SAP_IdenticalDatabaseSystem and also the Secondary Database under Replication Relations.
Manual Maintenance for SAP HANA < Rev. 102.02
If your SAP HANA system is older than revision 102.02 you won't see a relation here. In this case please add the secondary database manually by clicking on the button "Assign Secondary Database" in the table Replication Relations of Selected Database.
Please note that you have to manually adjust the direction of the dependency in case of a take-over.
SAP HANA DB which is the current Secondary System
For the secondary database system you see the role Secondary Database in LMDB.
On the view related databases you see the Primary Database under Replication Relations.
Please as for any other SAP HANA database make sure that technical instances and software components are delivered correctly on the respective views for the two physical databases.
Run Managed System Setup
For the managed system setup you can either start with the ABAP system or from the SAP HANA database reported by the SAP ABAP system (the virtual database). SOLMAN_SETUP will automatically recognize which databases belong to the scenario and select them accordingly.
Most of the steps are performed exactly as for a standard SAP HANA database managed system setup, you just do it for two databases at once instead of one.
In the step "Assign Diagnostics Agent" you will see that all hosts involved in the SR scenario are recognized.
In the step "Enter System Parameters" you have to make sure that you enter the correct hostnames. Enter all hostnames of the current Primary system that can take over the MASTER role for the Indexserver separated by comma. This way Solution Manager can always reach the database system.
Database Hostname Suggestion by SOLMAN_SETUP
Please double-check whatever hostnames SOLMAN_SETUP suggests and correct them if necessary.
Expert Step: DCNAME for Physical Databases
For SAP Solution Manager 7.1 SP 14 you have to perform another manual step, before you can fully monitor the SR scenario. With SAP Solution Manager 7.1 SP 15 this step is not necessary anymore.
Background:
As of now during the managed system setup the DB Connection Name is only maintained for the virtual database. This wouldn't be an issue, but SAP HANA metrics perform an additional validity check during activation, to make sure they only are activated for SAP HANA revisions that actually know this alerts to avoid grey metrics (as you know new checks are also implemented in HANA with every release). For a metric to pass this check the technical DB system needs a working DB Connection to be maintained. So when you now activate the monitoring for the SR scenario Solution Manager checks this DBCNAME parameter and notices that is doesn't exist for the physical databases and hence doesn't activate a huge amount of SAP HANA metrics based on HANA alerts.
Solution:
Unfortunately this DBCNAME cannot simply be maintained in the managed system setup for the physical databases, because Solution Manager doesn't let you access this databases separately. You have to maintain in parameter in the Generic Storage of SOLMAN_SETUP after you finished the managed systems configuration.
Call transaction SOLMAN_SETUP_ADMIN and go to "Generic Storage Admin". Select Configuration ID "RCA_SETUP-RCA_SETUP" from the drop-down. Filter the result table for the SID of your SAP HANA database. Make sure you filter in a way that you will find all three databases of the SR scenario.
From the list select the entry for: SYSTEM_ID = <Long SID of the virtual DB> ; SYSTEM_TYPE = DATABASE and copy the parameter value for the parameters DBCNAME and DB_USER.
Now select the entry for your physical database: SYSTEM_ID = <LongSID of the phycial DB> ; SYSTEM_TYPE = DATABASE. You will see that the DBCNAME parameter is missing and the DB_USER parameter is empty. Switch to Edit-Mode on top of the page and click "Add new entry". Add the entry for DBCNAME with the same value as the virtual DB and also maintain DB_USER.
Save your work. Repeat for the second physical database. You have to save and switch to Display-Mode before selecting the second physical database for it to work.
Please note that this is an expert transaction. Be very careful, do not play around and do not change any other entry than the ones described above!
In the next step you maintain the installation and log paths for both systems in the SR scenario.
Perform the rest of the Managed Systems Configuration as you would do it for a standard SAP HANA installation.
System Monitoring of SAP HANA with System Replication
The last step is the activation of the System Monitoring. From Solution Manager 7.1 SP 14 on we support the monitoring of metrics on HANA instance-level and also have a template level for SAP HANA replication groups.
Instance-specific metrics are basically metrics that can be specified "by host". These metrics were moved in a template "SAP HANA DB Instance" on instance-level. If you want to use this template please make sure that you use the template "SAP HANA DB (instance level separated)" on database system level. The template "SAP HANA DB (NEW)" contains all metrics, also the "instance-level" metrics, but they are configured as metric group that return one variant per host.
The template "SAP HANA DB Replication Group" can only be activated in the virtual database level. It contains system replication specific metrics. The template "SAP HANA DB (NEW)" also contains these SR specific metrics but they are shipped inactive. If you want to use only the database level template you can do this, just activate the SR specific metrics. They are easily identified by the prefix 'SR:'.
In the step "Define Scope" select the virtual database (the one for which you also ran the managed system setup). The two physical databases will be recognized automatically.
In the next step select the template combination you wish to use and activate the template assignment.
Result
After the activation the metrics will be collected and the DB replication group will be visible in System Monitoring. It is marked which site it the primary site and which one is the secondary site.
Metrics on Secondary Site
As you can see only there are no metrics collected on the secondary site, as a SAP HANA database cannot be reached if it is in stand-by mode. Only the metrics collected by the SAP Hostagent (the metrics on host level and some Availability metrics) are activated.
System Replication specific metrics can be found on virtual DB level.
Reconfiguration Tasks after SR Take-Over
We are working on enabling a complete automatic reconfiguration in the take-over case. As of now we are not quite there yet, so some manual tasks have to be performed.
This section is mainly then relevant for you if you don't plan to fail back immediately but have to run the database on SITEB (the new primary site) for some time. If it makes sense to run this reconfiguration or not always depends on how long SITEA will be out. During this time you cannot monitor any system though or will get false alerts.
Please note that you only will be able to continue monitoring the database SR group if system replication is re-established after the take-over. If you never re-establish system replication the new relationships will not be reported into SLD/LMDB and you will not be able to activate the monitoring for the secondary database. This doesn't mean your SITEA system must be running. You just need to register SITEA for system replication with SITEB. Then you should retrigger the SLD DS on SITEB (by toggling the parameter [ ] sld => enable). Now the new relationships are reported to SLD and you can begin the reconfiguration.
After a take-over SITEB will be reporting as the new primary site to SLD. To make sure LMDB, DBACockpit and System Monitoring are aware of this, some manual steps have to be performed currently.
Delete superfluous SAP_DatabaseReplicationDependency in SLD
Since the secondary database (new SITEA) doesn't report to SLD currently the SAP_DatabaseReplicationDependency created by it initially is never deleted. This leads to an additional superfluous SAP_DatabaseReplicationDependency association in SLD and hence in LMDB.
The easiest way to get rid of this, is to delete it directly in SLD.
Go to the new primary SAP HANA database in SLD and look for the relation SAP_DatabaseReplicationDependency with the role Dependent. Delete this association. It will be deleted automatically in the secondary database as well.
The LMDB will be updated automatically within 10-15 minutes after deleting the relationship in SLD.
Manual Maintenance for SAP HANA < Rev. 102.02
If your SAP HANA system is older than revision 102.02 the relation was not supported by the SAP HANA SLD DS and you had to maintain the relationship manually in LMDB.
Please remember to manually adjust the direction of the dependency in case of a take-over.
Correct SAP_IdenticalDatabaseSystem in LMDB
Before the take-over the virtual database points to SITEA as identical database. You will have to correct this relationship after the take over.
If you have coincidentally set the parameter sql_connect_hosts to the correct logical host in SITEA and SITEB as described earlier, in SAP HANA rev. 100 to 110 this association will be corrected automatically. As of SAP HANA rev. 110 the parameter SLDVIRTDBHOME will be considered.
Open the virtual database in LMDB and switch to the Related Databases view.
Delete the old relation first.
The assign the correct physical database.
Save your changes.
Adjust DBACockpit DB Connection
As of now the DBACOCKPIT DB connection is also not corrected automatically. When we first set up the connection we used the hostnames of all hosts that can assume the MASTER indexserver role in the at this time primary database SITEA. Now we have to change this to the hostnames of all hosts that can assume the MASTER indexserver role in SITEB.
Open the managed system setup for the virtual database and switch to the step "Enter System Parameter". You will see that the DBA Cockpit connection status is red.
Adjust the database connect string with the correct hostnames and save.
You can check the DB connection in DBACOCKPIT or you can also use report ADBC_TEST_CONNECTION.
Warning during DB Connection Setup
If you get the warning "DBA cockpit connection <DBCNAME> cannot be established" please check the details in the log. If the error message is "Connection failed (RTE:[89006] System call 'send' failed, rc=32:Broken pipe)" check that the HDBUSERSTORE is maintained for the HANA system. I
f this is the case, check if the DB connection is working directly in DBACOCKPIT.
If yes, close and reopen the managed system setup. The DB Connection should show green now.
Rerun Monitoring Activation
Now you can rerun the System Monitoring activation. Go to the System Monitoring setup step "Define Scope" and select again the virtual database. Go to the next step. You don't have to change any template assignments here. All you have to do is click "Apply and Activate => All managed objects"
Now the monitoring shows the correct SR role assignments and also monitor the values of the new Primary database and the SR specific metrics.
Troubleshooting
If you have problem with the managed system setup please refer to the Troubleshooting section of this wiki.