Page tree
Skip to end of metadata
Go to start of metadata

Purpose:

The purpose of this page is to enable you to understand the process involved in managing persistence of Process Integration (PI) or Process Orchestration (PO) messages (we will just use the term PI from here on), the various Background Jobs that are involved, how different types of messages are persisted and the various parameters which impact on the removal of messages from the PI DB tables.

Overview: 

The persistence of messages in a PI system is a complex situation and requires inputs from all stakeholders involved. This can include, the PI Developer/functional teams, the Basis Admin team, Database Admins and very importantly, the Business stakeholders. How you manage the persistence will depend on several factors which are generally unique to your business organization. This requires careful planning which ideally is taken care of prior to the go-live of any system or any new additional interfaces. It is an important factor when sizing a PI system e.g. estimating the size and volume of PI messages as well as the number of versions of each message you plan to persist and the length of time you need to persist these for. 

Develop your strategy:

  • If you have requirements to configure additional persistence for PI messages e.g. using the Logging or Staging functionality, it is recommended to do this only for interfaces which have a specific business requirement
  • Do not configure Global Logging or Staging properties, which impact all interfaces, if there is no specific business requirement to do so
  • The following factors should be taken into account:
    • the specific interfaces which require additional persistence
    • the type of interface (synchronous or asynchronous) - different persistence settings are possible for both
    • the volume of messages to be processed by the interface
    • the estimated size of the messages
    • the number of versions of each message you will be persisting (depending on Logging or Staging configurations, multiple versions of each message can be persisted to the DB)
    • the length of time these need to be persisted for
    • whether you need to move the messages to an Archive store or whether the Delete Job is sufficient to meet your business requirements
    • if you do identify a need to move the messages to Archive, then identify which specific interfaces require this
    • generally, there is not a requirement to Archive every version, of every message of every interface so it is important, for overall system resource consumption, to minimize the amount of data which is sent to archive i.e. only archive the messages which have a specific business requirement to do so
    • note: if you are configuring a custom Archive Job, it is really important that you use the Rules option to specify messages from particular interfaces to send to the Archive. If you configure a custom Archive Job with no Rules, then every version, of every message of every interface is sent to the Archive store and the Delete Job

Understand the main PI related tables and parameters related to message persistence:

Asynchronous message persistence:
xiadapter.inbound.persistDuration.default 86400000 (default is 1 day)
xiadapter.outbound.persistDuration.default 86400000 (default is 1 day)

Synchronous message persistence:
messaging.syncMessageRemover.removeBody true (default is not persisted - approx 1000 messages stored in memory cache and oldest message flushed out when new sync messages are processed)
messaging.log.retainTime 7;

Audit Log persistence:
messaging.auditLog.memoryCache true;
messaging.orphanAuditLog.retainTime 30 
messaging.auditLogEnabled true

Logging/Staging persistence: 
xiadapter.logger.conf
xiadapter.stage.conf    MS=3

Read the below resources:

  1. SAP Blog Introduction of the tables that keep message related data in the database of PI/PO system

  2. KBA 2592310 PO/AEX Messaging tables relevant for DB reorg jobs

  3. KBA 1872758 How XI/PI messages are persisted in Adapter Engine
  4. SAP Note 1760915 FAQ: Staging and Logging in PI 7.3 and higher

Troubleshooting from PI perspective:

The following checks should be made:

  1. Check the Background Job Processing Monitor for any issues related to the Delete Job (or Archive Job if this is also configured)
  2. Check the *.persistDuration and Logging and Staging parameter settings (see table below) - Logging and Staging parameter settings can result in multiple versions of every message from every interface being persisted to the DB. This can but strain on DB and system resources and these configurations should be put in place after careful planning and testing. 
  3. Check if there are any specific Retention settings for individual interfaces configured in the NWA → SOA → Monitoring → Message Retention
  4. Check the Number of Entries in the main PI related tables - go to NWA → Open SQL Data Browser
    BC_MSG
    BC_MSG_LOG
    BC_MSG_VERSION
    BC_MSG_AUDIT
  5. Check the status of the messages in BC_MSG and BC_MSG_LOG - only final status (DLVD or FAIL) messages are removed by the Delete/Archive Jobs - some sample SQL statements are documented in PI Adapter Engine Troubleshooting SQL Statements
    ***Note: making DB level changes to PI message can render your system in an unsupported state. Do not make any changes to PI messages at DB level.

Depending on your specific operational requirements:

  1. the IdocDBTableCleanupPeriodicAdvanced job should be configured as per note 2624188 IdocDBTableCleanupPeriodicAdvanced: a new improved job for IDoc_AAE data deletion with optimized option for SAP ASE DB ***ensure the HandleMSEvent parameter is set to false as described in the note
  2. the B2BAckStorageCleanJob job should be configured as per note 2515397 B2BAckStorageCleanJob to clean orphan B2B acknowledgment report data

Abnormal Growth in PI related DB tables: 

If you observe abnormal growth in PI related DB tables take the following actions:

  1. Verify if the Background Jobs are running as expected i.e. no errors are reported. Go to the NWA → Background Job Monitor
  2. Check if additional Global Logging and/or Staging parameters are configured - if so, revert to default (if possible).
  3. Check if Interface (ICO) level Logging/Staging parameters are configured.
  4. Check the number of entries in PI related DB tables
  5. If nothing unusual is found on the PI side, then you may need to investigate further from the DB perspective. Tables with high I/O need regular reorgs from the DB perspective. Each DB vendor may have different handling in this regard. Check with your DBA if further assistance is required.

Parameter Glossary:

XPI Service: XI AdapterDefaultCategoryDescription

xiadapter.stage.conf

MS=3PersistenceGlobal Staging parameter

xiadapter.logger.conf

" "PersistenceGlobal Logging parameter

xiadapter.inbound.persistDuration.default

86400000(ms)RetentionPersists asynchronous inbound messages for 1 day

xiadapter.outbound.persistDuration.default

86400000(ms)RetentionPersists asynchronous outbound messages for 1 day
messaging.syncMessageRemover.removeBodytruePersistenceProperties Related to Synchronous Messages
messaging.log.retainTime7 (days)RetentionProperties Related to Synchronous Messages

***In general we recommend not to change Logging/Staging Globally since usually not every interface has the same requirements. Instead the configuration should be done individually per interface based on the business requirements for this interface.
***With SAP Note 1960448 Configure message retention period per interface, you can configure interface specific retention times to suit needs of specific interfaces. See Configuring Message Retention

XPI Service: MessagingDefaultCategoryDescription
syncMessageRemover.removeBodytruePersistence (synchronous)

The messaging system removes the payload of synchronous messages by default as soon as they reach a final state.
This feature can be temporarily disabled to allow the message body to be analyzed in the case of errors (also for synchronous messages) by using this property.
This property must only be changed for extended error analysis. Keeping all message bytes of synchronous messages in the memory can lead to OutOfMemoryErrors in AS Java.
You can modify this property online.

messaging.syncMessageRemover.messageCount1000Persistence (synchronous)

This property defines the maximum number of synchronous message headers that are kept in memory for monitoring and duplicate detection purposes.
The synchronous-message-remover job checks how many messages are in final state and could be completely removed from memory.

Then it removes the oldest synchronous message headers until a maximum number of message headers are left in memory, as configured by the messageCount property

You can modify this property online..

messaging.auditLogEnabledtruePersistence (Audit Log)This property enables or disables the audit log.

Resources: 

SAP Note 2014949 Removing orphan audit log entries
SAP Note 1960448 Configure message retention period per interface

SAP KBA 2442344 How to enable logging for synchronous messages

SAP Help Background Job Types
SAP Help Data Storage Security for the Advanced Adapter Engine

SAP Community:
Blog Message Staging and Logging Options in Advanced Adapter Engine of PI/PO 7.3x/7.4
Blog Archiving & Deletion of messages in PI – Part 1

Wiki Persist duration time at the Adapter Engine for synchronous and asynchronous messages

https://answers.sap.com/questions/12061395/staging-vs-logging.html