Possible memory leaks in 4.4.2

[theleo_ua]

GZDoom 4.4.2 x64
Windows 7 Ultimate x64
Geforce GTX 770 2GB


Possible memory leaks


Steps to reproduce:

1) download testmod (now this link without iwad): https://u.pcloud.link/publink/show?code ... 0rcyiMX9WV
2) Run "001__GZ_3_3_2__5GB\_TESTMOD.BAT" with GZDoom 3.3.2 and hexen.wad iwad
3) Press "ESC / New game / skill 5"
5) Wait till loading finished
6) Check gzdoom.exe memory usage: https://u.pcloud.link/publink/show?code ... nXcHPn07Qk
(5.7 GB)

7) Run "002__GZ_4_4_2__8GB\_TESTMOD.BAT" with GZDoom 4.4.2 and hexen.wad iwad
8) Press "ESC / New game / skill 5"
9) Wait till loading finished
10) Check gzdoom.exe memory usage: https://u.pcloud.link/publink/show?code ... tRybl9YEqV
(8 GB)

Actual result: 5GB for 3.3.2, 8GB for 4.4.2

Expected result: 5GB for 3.3.2 and 5GB for 4.4.2


NOTE: the reason I created this report and thinking it's a memory leak, because I have another heavy mod, and played it before 4.4.2 without much lags for loading graphics during game. But when I tried 4.4.2, I noticed big amount of lags (I can create and upload video comparisons, if you want). Trying to figure out, why those lags appeared, I noticed next:

a) my mod in gzdoom 4.4.2 can now consume 8GB of RAM and this is not the limits (I have test cases, how to create even 16GB of RAM usage, but not tried them)
b) I noticed next change in 4.4.2 changelog details: rewrote the hardware texture precacher to use the new texture management to properly track the data to delete.

So I start thinking, that GZDoom should use less RAM, than in my examples (and there should not be much lags, like in previous GZDoom versions). Thats why I think it's leaks

If you need my heavy testmod for 16GB RAM usage, and also video examples of scenarios to consume that amount of memory, please tell me, I will create and upload this mod. I hope testmod I uploaded above is enough, but if not, I will upload another one

NOTE 2: be sure to use my config files (using of them are included to bat files in testmod), because precaching and XBRZ4x should be set to on (and also maybe some other options, which I missed)

[drfrag]

There's a memory leak in 4.4.2 and it was fixed. I think this is a duplicate of viewtopic.php?f=7&t=69055
Try a recent devbuild to see if the problem persist: https://devbuilds.drdteam.org/gzdoom/

[theleo_ua]

There's a memory leak in 4.4.2 and it was fixed. I think this is a duplicate of viewtopic.php?f=7&t=69055
Try a recent devbuild to see if the problem persist: https://devbuilds.drdteam.org/gzdoom/

reproduced in gzdoom-x64-g4.5pre-125-g58f5c030f.7z (both in attached testmod and in my heavy mod) : https://u.pcloud.link/publink/show?code ... gom0ctv8HV

[drfrag]

I've noticed that ram usage is generally much higher in GZDoom than in LZDoom (based on 3.3.2). That doesn't mean there's a memory leak, a memory leak means ram usage keeps increasing until your system freezes (if you're not quick enough to press alt+f4 that is). But certainly if it's confirmed on high ram scenarios it's an issue anyway.
I asked a guy at discord to compare usage between versions to confirm it but he didn't reply and i only have 4 GB of ram. Have you texture precaching enabled in the case of 4.4.2 and disabled in 3.3.2 by any chance?

[_mental_]

There were so many changes between 3.3.2 and 4.4.2. Could you please narrow it down? I would start comparison with 3.7.2 and 4.0.0 first.

[theleo_ua]

hexen.wad is removed from testmod (sorry for this), link is updated

Have you texture precaching enabled in the case of 4.4.2 and disabled in 3.3.2 by any chance?

gl_precache=true (in all versions/testmods)

There were so many changes between 3.3.2 and 4.4.2. Could you please narrow it down? I would start comparison with 3.7.2 and 4.0.0 first.

3.7.2 (still 5 GB): https://u.pcloud.link/publink/show?code ... UouJiAkY8y

4.0.0 (still 5 GB): https://u.pcloud.link/publink/show?code ... cOvjSWGN7y

I've noticed that ram usage is generally much higher in GZDoom than in LZDoom (based on 3.3.2). That doesn't mean there's a memory leak, a memory leak means ram usage keeps increasing until your system freezes (if you're not quick enough to press alt+f4 that is)

I understand you and you are right (this may be my mistake too), but what about this scenario:

1) Run "002__GZ_4_4_2__8GB\_TESTMOD.BAT" with gzdoom-x64-g4.5pre-127-ge740e00e0 (or GZDoom 4.4.2) and hexen.wad iwad
2) Press "ESC / New game / skill 5"
3) Wait till loading finished
4) Check gzdoom.exe memory usage: 7 852 864 MB / 8 045 584 MB
4.4.2 (bottom part of the screenshot): https://u.pcloud.link/publink/show?code ... a4HH6Dz4I7
4.5pre-127-ge740e00e0 (top part of the screenshot): https://u.pcloud.link/publink/show?code ... hCG7fNXlxV

5) Repeat step 2 several times (for example 10 times)
6) Check gzdoom.exe memory usage: 8 001 304 MB / 8 201 284 MB
4.4.2 (top part of the screenshot): https://u.pcloud.link/publink/show?code ... a4HH6Dz4I7
4.5pre-127-ge740e00e0 (bottom part of the screenshot): https://u.pcloud.link/publink/show?code ... hCG7fNXlxV

Actual result: memory usage increases with each map restart

Expected result: memory usage should not increase with each map restart

Can this be considered as memory leak?

Also maybe my nvidia drivers and nvidia driver settings are make some influence on this? If you need, I can tell my driver version and post sreenshots with settings

But certainly if it's confirmed on high ram scenarios it's an issue anyway.

Agree. Also, could somebody explain, why 30MB pk3 can consume so much RAM? Is it really 8GB uncompressed data are compressed to 30MB zip file? (UPDATE: XBRZ4x takes 16 times more memory, so 500 MB instead of 8 GB, and it seems to be ok for uncompressed 30MB pk3)

i only have 4 GB of ram

You can turn off XBRZ4x and continue testing, while GZDoom will consume 16x less RAM. I tried this, and now GZDoom will take 500MB instead of 8GB, but scenario above (with increasing memory after map restart) will still work

[drfrag]

Those ini files you sent are different, you're using gl_texture_hqresizemode=4 and gl_texture_hqresizemulti=4 for the 8 GB folder and not for the 5 GB folder.
So they are not the same settings and it's logical that the engine uses more ram with hqresize enabled. Besides those two engine versions (4.4.x) are very similar.
The increase in ram usage when loading maps doesn't mean there's a memory leak, there's a garbage collector and that ram should be freed later.
Also texture precaching is different between 3.3.2 and 4.4.2. If there's a higher ram usage without precaching and hqresize that's a different story.

[theleo_ua]

Those ini files you sent are different, you're using gl_texture_hqresizemode=4 and gl_texture_hqresizemulti=4 for the 8 GB folder and not for the 5 GB folder.
So they are not the same settings and it's logical that the engine uses more ram with hqresize enabled. Besides those two engine versions (4.4.x) are very similar

3.3.2 engine doesn't have 2 options (one for filter and another for value) - it has single option (values are XBRZ2x, XBRZ4x etc)

4.4.2 engine has 2 options (one for filter and another for value) - it has 2 options (filter type, values are XBRZ HQx etc, and value, values are 2x, 4x)

That's why my config for 3.3.2 doesnt have gl_texture_hqresizemode and gl_texture_hqresizemulti. But - if you use my 3.3.2 config for 3.7.2 or 4.0.0. or even 4.4.2, you will see, that gzdoom will add gl_texture_hqresizemode=4 and gl_texture_hqresizemulti=4 - exactly with values 4 and 4. So adding options gl_texture_hqresizemode=4 and gl_texture_hqresizemulti=4 to 3.3.2 ini makes no sense, because engine doesn't know them

Anyway, 3.7.2 and 4.0.0 with gl_texture_hqresizemode=4 and gl_texture_hqresizemulti=4 in INI, has same 5GB ram usage.

If you want, I can add gl_texture_hqresizemode=4 and gl_texture_hqresizemulti=4 to 3.3.2 INI and try 3.3.2 again, but I bet result will be the same (5GB)

The increase in ram usage when loading maps doesn't mean there's a memory leak, there's a garbage collector and that ram should be freed later

ok, thanks

Also texture precaching is different between 3.3.2 and 4.4.2. If there's a higher ram usage without precaching and hqresize that's a different story.

ok, I will try with "precache = off" and "hqresize = off", and will tell the results

[theleo_ua]

Also texture precaching is different between 3.3.2 and 4.4.2. If there's a higher ram usage without precaching and hqresize that's a different story.

Tried 3.3.2, 4.4.2 and 4.5pre-127-ge740e00e0 without precaching and hqresize - here are the results: https://u.pcloud.link/publink/show?code ... 7OEBV9JBwy

Should I try with 3.7.0 and 4.0.0 ?

[drfrag]

I've tried it myself, 309 MB for a recent master and 239 MB for the old lzdoom branch. Not much of a difference with this mod. 893 MB without precaching and 1060 MB with it seems there's more initial overhead (but i've seen a higher difference with other mods such as PB). I can't test with 4x becouse i can't run anything using more than 2 GB.
I've just ran the mod without your inis, let's see if someone else can reproduce the issue but i'm not sure you're using the same settings. Why don't you try with the same ini for both 4.4.0 and 4.5pre?
I forgot the file was for 3.3.2 even when the folder is named 3.3.2, i thought i was expected to try 4.4.0 now.

[theleo_ua]

I've tried it myself, 309 MB for a recent master and 239 MB for the old lzdoom branch. Not much of a difference with this mod. 893 MB without precaching and 1060 MB with it seems there's more initial overhead (but i've seen a higher difference with other mods such as PB). I can't test with 4x becouse i can't run anything using more than 2 GB.
I've just ran the mod without your inis, let's see if someone else can reproduce the issue but i'm not sure you're using the same settings. Why don't you try with the same ini for both 4.4.0 and 4.5pre?
I forgot the file was for 3.3.2 even when the folder is named 3.3.2, i thought i was expected to try 4.4.0 now.

I'm ready to exactly do what you want, but I need to exactly understand the steps, which you want. Let me try to guess:

1) Take gzdoom.ini from 4.4.2 from testmod (but turn off HQresize and texture preaching)
2) Use ini from step 1 and try gzdoom 3.3.2 with it
3) Use ini from step 1 and try gzdoom 3.7.2 with it
4) Use ini from step 1 and try gzdoom 4.0.0 with it
5) Use ini from step 1 and try gzdoom 4.4.2 with it
6) Use ini from step 1 and try gzdoom 4.5.pre with it

Those 6 steps are exactly what you want me to do?

[drfrag]

No, i mean just use the same ini with your settings for both 4.4.0 and 4.5pre. You said there was a big difference for both versions.

[theleo_ua]

No, i mean just use the same ini with your settings for both 4.4.0 and 4.5pre. You said there was a big difference for both versions.

if you mean these my quote:

NOTE: the reason I created this report and thinking it's a memory leak, because I have another heavy mod, and played it before 4.4.2 without much lags for loading graphics during game. But when I tried 4.4.2, I noticed big amount of lags (I can create and upload video comparisons, if you want). Trying to figure out, why those lags appeared, I noticed next:

then I mean 4.3.3 (I never tried 4.4.0 with my heavy mod). Here are examples of my last 9 tries with my heavy (hhmix) mod: https://www.youtube.com/playlist?list=P ... lymer=true (first 2 videos in playlist are played in 4.2.4, and last 7 videos are in 4.3.3). There are no any often lags like "1 lag per 10 seconds" and no any longtime lags like "freezing game for 10 seconds" (during all 9 longtime videos)

But if I will try the same mod now (with 4.4.2), even in easiest map01 from hexen.wad iwad, there will be often lags like "1 lag per 10 seconds" and also longtime lags like "freezing game for 10 seconds" (I can record a video example, even in 4.5.pre, if required)

So - here are the test results for 4.3.3, 4.4.0 and 4.5.pre (both with HQ/cache and without) with same INI from 4.4.2: https://u.pcloud.link/publink/show?code ... L4k7G60hbX (tests for attached 30MB testmod, not for heavy hhmix of course)

[theleo_ua]

You said there was a big difference for both versions.

So - I tried my heavy hhmix mod on 4.3.3 and 4.5 pre with same INI from testmod (from 4.4.2) on map01 and map02 from hexen.wad iwad. And there is huge difference in lags and memory usage. No any noticeable lags in 4.3.3, and low ram usage (500 MB for map01 and 700 MB for map02+map01), but a lot of very noticeable and "almost unplayable" lags in 4.5 pre and high ram usage (1GB for map01 and 3 GB for map02+map01)

If you want, I can create video comparison with my voice comments, where it lags (with displaying gzdoom memory usage in realtime), and if you consider this video useful for further tests and explorations, I can upload mod (if required)

[_mental_]

Regarding lags, could you please compare both versions with the following stats enabled in console?

Code: Select all

stat rendertimes
stat renderstats
stat think