SAP system is started by startsap script. startsap script calls sapstartsrv (via sapcontrol), which then starts the SAP system. Therefor the operating system limits used by SAP system will be the same as are known by sapstartsrv process. This is described in SAP note
1437105 Operating system limits for SAP instances
To set the limits for all sapstarstrv daemons that were started using the sapinit script, you can set the limits in /usr/sap/sapservices:
limit.descriptors=<value> limit.stacksize=<value> limit.datasize=<value>
Check the limits used by SAP via RSBDCOS0 report (resolution 2. of SAP note 1563583). Data limit has to be "unlimited".