New in DataServices 4.1 is the RFC transport method. When using this we do no longer write a file on the SAP side, instead we collect a few rows and then call a remote function call, the host of this function is the al_engine process. So every time the ABAP collected 5000 rows it will send the data in binary format to the al_engine.
This approach has multiple advantages
- Easier to setup as you do not longer need to ask for a directory on the SAP server and permissions to read that somehow
- It is faster as we figured out. When writing the file we had to convert decimals to strings and other things and we found out that writing the file is actually slower than transfering the data in a binary format.
- Faster due to parallel processing. With files the DataServices engine had to wait until all data was collected in a file and once transferred it started tprocessing the data and while processing the SAP system had nothing to do. Now it all happens in parallel, while the SAP system is collecting the next batch DataServices is already processing the current rows. So quite often you will see that before the ABAP ran e.g. 10 minutes and the engine additional 5 minutes, so 15 minutes overall, now in the 10 minutes of the ABAP the engine did its part as well, so 10 minutes overall.
Load balancing can be achieved with the RFC Data transfer method when the datastore setting "Execute in 'background'" set to no. "If Execute in background" is set to yes, Data Services al_engine makes the connection to whichever server the ABAP is executed on, but it has to send the data back through the server listed in the Datastore setting "Database Server".
For RFC transport method to work, the ABAP program has to be regenerated. For development (datastore set to generate_and_execute) no problem, it does regenerate the ABAP every time you execute anyhow. But for prod, the ABAPs have to be re-generated in dev/test first and next - as usual - transported to production in order to be used in execute_preloaded mode.
The one thing you should do (for execute_in_background=yes required) is going into the SAP transaction /nsm59 and create a new RFC destination of name SAPDS as Connection Typ T (TCPIP) and under Technical Settings set it to Registered Server Program and the Program ID to SAPDS as well.
The connection test will fail, this is expected behavior as al_engine has not started an RFC Server to communicate with, and the program ID will be SAPDS_XXXXXXXXXX. Using additional information passed to the ABAP program each started ABAP program will know what al_engine process belongs to it, so running multiple ABAP jobs simultaneously will work fine.