It is possible to limit the size of trace files using the parameter rdisp/TRACE_LOGGING.
Details about how to setup this parameter in SAP note 112.
112 Trace and error information in the "dev_" files
With SAP kernel 72X the traces are rotated by specific triggers that occurs in specific time intervals, however, these time intervals may vary depends of the component or even the action a work process is doing. In some specific situations the trace can increase further than the size set in the parameter.
With 74X kernel the rotation are much more precise avoiding the traces to grow much more than the specified.
See also:
1370207 - Creating work process trace files
2020096 - How to increase sapstartsrv trace level and limit the trace file size