Basically, when the game is paused, GC collection slows down to a crawl. This means that, if a normal menu allocates classes constantly, the GC never collects memory fast enough to offset the constant increase, and you end up with a net increase in RAM usage.
You can reproduce this with Nash's PDA starter kit:
- Clone the repo or download the source Zip and launch GZDoom while using the entire folder as a file, with Doom 2 as the IWAD.
- Start the game and do "stat gc" in the console.
- Pick up the items in front of you, then press the Use Inventory key on the item you get.
- Click any of the entries in the list.
- Memory allocations will now be too fast for the GC to keep up with. Once "Alloc" reaches "Thresh", it'll be stuck in the Sweep phase.
Doing "gc pause 1; gc stepmul 400000" in the console also seems to help while the game is paused.
In general, this is a pretty serious issue that needs to be fixed. There's no reason for GZDoom to slow down the GC like this when the game is paused.