There are several ways to check the current memory consumption of a running query. In the transaction ST02 you can have a list displayed where you can see how much memory workprocesses (assigned to a user) allocate: ST02 -> button 'Detail Analysis Menu' -> SAP Memory -> Mode List
In this case 667 MB extended memory and no heap memory is allocated (see also Memory Parameters). With 'Enter' it is possible to refresh this list.
In many cases it is helpful to stop in the debugger in order to determine the memory consumption at a certain processing step. In this context the most important default breakpoint is called 'Data Transfer OLAP Processor / Frontend Data Get' which you can choose when running the query in RSRT (Execute&Debug):
Once you stop in the debugger please press F7. If the query has characteristics in the rows and columns (cross list), it is necessary to do this three times. So after the F7 again F8, F7 and F8,F7 (the processing block 'Data Transfer' called three times). Then you are at the stage in the query processing where the OLAP engine transfers the data to the frontend (see Query Performance and Memory Consumption). You can now run the transaction ST02 to check the memory consumption or you can do this in the debugger itself: