It is possible to limit the size of trace files using the parameter rdisp/TRACE_LOGGING.
Details about how to setup this parameter in SAP note 112.
With SAP kernel 72X the traces are rotated by specific triggers that occurs in specific time intervals, however, these time intervals may vary depends of the component or even the action a work process is doing. In some specific situations the trace can increase further than the size set in the parameter.
With 74X kernel the rotation are much more precise avoiding the traces to grow much more than the specified.