In any software design apart from meeting the functionality the success factors, among many, also consists of performance, modifiability, scalability and reliability.
It will be some of these parameters on which I would like to present my thoughts about Data Orchestration Engine. The irony about these factors being there absence is easily recognizable than their presence. More often than not effort needs to be put to explore these aspects. Or these factors surface only when the user confronts the situations.
Considering the case that NetWeaver Mobile7.10 (Data Orchestration Engine) started itself with the demand to meet the needs to handle thousands of receivers in an enterprise, performance formed the primary requirement. It could be simply unacceptable if the receivers need to take hours to synchronize their updates. Or the data fetch from ERP to DOE taking days. The volume scenario with context of performance becomes critical in situation like business reorganization or year-end changes etc. The updates in middleware like DOE for such scenario are not supposed to be bottleneck case. But rather are taken as opportunity to prove its performance. The cases which were taking days previously are now getting completed in hours with DOE.
The design factors that helped in achieving this are related to designing core pillars as separate concerns: data consolidation, distribution, extraction and synchronization.
And, of course, parallel processing.
Provided the system & landscape configuration:
- The receivers in parallel can perform synchronization, submit their updates to DOE and fetch the data prepared for them by DOE. The processing of updates submitted by receivers happen separately as batch job and does not block the synchronization process.
- The data consolidation, from ERP or backend system to DOE, for multiple objects can be done in parallel.
- Having the data consolidated in DOE the mapping of data to relevant receiver for different object occurs as separate job.
- Preparation of receiver relevant content based on mapped information occurs as a separate job.
- Processing of receiver's update, the consolidation of data, proper receiver to data mapping and preparing required content for receiver for next synchronization are designed to happen in parallel with each other.
With such parallel processing, each as separate job, DOE can be made to exploit the resource not only in isolated system but even in the cluster or distributed landscape as well.
Here will take the approach mostly in terms of extensibility, customization & configuration options available with DOE.
The extensibility in terms of support for different content delivery to receivers matching receiver capability is possible by way of custom channel. Based on the channel implemented and configured for a receiver, DOE can deliver the content through the relevant channel. Examples of various such implemented channels are: SMS, RSS, native web service implementation, .Net based implementation, J2ME based implementation.
Though BapiWrappers are meant to transfer data to & fro with ERP system, there is no restriction that these need to be for Bapi's only. These wrappers can communicate data from different source say through web service call, or any other adapter.
The customization in terms of fetching or updating the data to ERP systems is possible by way of implementing various BAdI exits possible for BapiWrappers.
In a way one can consider the custom channel and bapiwrapper as a means to adapt to Data Orchestration Engine platform as well.
Also customization in terms of handling the content during their consolidation process is provided by way of custom services. These are the services that the application developer can include in the standard flow of data object processing in the DOE.
Customization in terms of when to (re)calculate the mapping of data to receiver can be done by way of introducing custom job as the rule execution.
Configuration per se DOE provides various options in terms of:
a- when to prepare of contents for receivers from mapped content
b- Which all rule to keep active for mapping data to receiver
c- Performance impact
d- Other system related parameters
Factors that affect the scaling any system can be broadly divided into two:
a- processing capability
b- Storage or I/O capability
In terms of processing capability, as most of the jobs are parallel taking committed data independently, it can be scaled by configuring number of parallel jobs based on CPU. The only inhibiting factor that prevent full blown parallelization and exploiting processing capability is to ensure sequential processing of any given record, even if as a separate job, for a given data object in terms of consolidation, receiver mapping and final content preparation (with main factor being receiver mapping to data ). However within the functional constraints to be met, system can scale in terms of processing.
The I/O, however, is critical factor affecting various stages viz data consolidation, mapping, storage of prepared content for receivers. The I/O capability thus determine how much can DOE be scaled.
As a measure of reliability, the content taken and processed by most of the jobs is ensured to be in committed state. If the job fails it can simply be restarted. And the dependent job will only pick from the committed data from previous job.
Also DOE ensures recovery option for most of landscape component failure (Refer: SAP NetWeaver Mobile 7.1 Backup and Recovery ).