Unclearable and constant SSAO lag

Moderator: GZDoom Developers

Accensus
Posts: 2377
Joined: Thu Feb 11, 2016 9:59 am

Unclearable and constant SSAO lag

Post by Accensus »

Disclaimer: posting this on behalf of a buddy, so I don't know too much about the mod's internals, only what happens and when.

Tested on g4.5pre-61-g4de8851e0 x64. Can only reproduce with Vulkan. Tested with OpenGL but couldn't get it to lag. It might not be impossible, just really, really difficult to do so.
Mod author reports the same issue and according to him it has been present for quite a while. If I understood correctly, even back in 4.3.3.

There are no exact steps because it always seems to be random, but the closest I can come to those is:
1. Start a map, any map. I've had it happen on E1M1 and MAP01. Tested the most on E1M1.
2. IDFA/give all to get all weapons
3. Empty the BFG into a wall. Try shooting the listed weapons in various directions for a better chance of lag.
4. Empty a mag from the rocket launcher (slot 5).
5. Empty the entire flamethrower (slot 5). Use alt-fire as well as that seems to be a good trigger.

NOTE: If it doesn't lag, go to step 2. It's bound to lag. Haven't had to IDFA more than twice before it does.

By the time you're done, "stat gpu" should show about 30 ms for SSAO. It doesn't go up gradually. One moment it's 1 ms and the next it's 30-50 ms. This is NOT cleared when switching maps, which is the main reason I'm reporting this and not writing it down as a user error. The only way to get rid of it is to either restart the game or disable SSAO. Re-enabling it again starts lagging everything down to 4 FPS. It gets worse if you look at outdoor areas like E1M1's blue armor secret.

The mod is private, so link to DL the example will be sent to the developers via PM shortly.

NOTE: The mod is going to take a while to load. It ramps up my RAM usage to about 2 gigs before it goes down to ~800 MB. Starting a level is also going to take about 4-5 seconds.
Accensus
Posts: 2377
Joined: Thu Feb 11, 2016 9:59 am

Re: Unclearable and constant SSAO lag

Post by Accensus »

Update: Okay so the lag seems to be because I ran out of VRAM. I was using Trilinear with Normal6x and the mod ate 8 gigs of VRAM in no time.
User avatar
phantombeta
Posts: 2009
Joined: Thu May 02, 2013 1:27 am
Graphics Processor: nVidia with Vulkan support

Re: Unclearable and constant SSAO lag

Post by phantombeta »

I'd like to mention that running out of VRAM with Vulkan (at least on Win 10 with a 10-series Nvidia GPU) seems to be a complete crapshoot of undefined behaviour. In some rare times it'll crash with an "out of device memory" error, but most often it'll use system RAM as swap. (Somehow. Even though supposedly the library GZDoom is using doesn't handle swap)
In some rare cases it'll corrupt the shit out of everything. Which can be pretty terrifying:




Edit: I should probably also mention: That corruption is... Pretty hard to reproduce. I can't ever seem to trigger it on purpose. I've also had the screen (across both monitors) "flash" when it triggers. Maybe it has something to do with Windows? Maybe it only happens once every time you boot Windows...
User avatar
Rachael
Admin
Posts: 13084
Joined: Tue Jan 13, 2004 1:31 pm
Preferred Pronouns: She/Her

Re: Unclearable and constant SSAO lag

Post by Rachael »

Considering that drivers are ring-0, the corruption sounds like memory corruption caused by the drivers, themselves. Your system is likely in an unstable state when this happens - if you see this corruption at any time, then when you finish your playing you should restart Windows just to be on the safe side.
User avatar
Graf Zahl
Lead GZDoom+Raze Developer
Lead GZDoom+Raze Developer
Posts: 48325
Joined: Sat Jul 19, 2003 10:19 am
Location: Germany

Re: Unclearable and constant SSAO lag

Post by Graf Zahl »

This is a known issue with Vulkan on NVidia. I have no idea what kind of weed they were smoking when adding out-of-VRAM workarounds to their internal allocator but all this produces is instability in client code. If the hardware runs out of VRAM it needs to report an error, not do it the OpenGL way any keep things running at all costs. This has been one of the biggest issues with OpenGL because it won't allow a client to recognize and deal with memory pressure issues, leading to bad performance.

And TBH, if they do not fix this soon it may cause them some very serious trouble in the future when the word gets around that AMD is simply better for modern games.
dpJudas
 
 
Posts: 2890
Joined: Sat May 28, 2016 1:01 pm

Re: Unclearable and constant SSAO lag

Post by dpJudas »

It isn't that simple. On modern OSes the GPU memory is a shared resource between multiple applications and as a result the memory can always risk being swapped to system memory. All we see here is a poor implementation on Nvidia's behalf.

Fat chance on AMD being considered the better option for modern games. They haven't been able to produce a competitive GPU for years now.
User avatar
phantombeta
Posts: 2009
Joined: Thu May 02, 2013 1:27 am
Graphics Processor: nVidia with Vulkan support

Re: Unclearable and constant SSAO lag

Post by phantombeta »

Rachael wrote:Considering that drivers are ring-0, the corruption sounds like memory corruption caused by the drivers, themselves. Your system is likely in an unstable state when this happens - if you see this corruption at any time, then when you finish your playing you should restart Windows just to be on the safe side.
Unlikely. This has never caused any system instability on my end, everything works just fine after this happens. No other programs break like this after this happens to GZDoom, not even ones that use hardware acceleration. (e.g., Firefox, Discord, Steam, GOG Galaxy...)
(It's not the first time I've had it happen, and I've never restarted because of it. I don't restart often either, so there's no way this wouldn't have caused problems if it really were actual system RAM corruption.)

Based on how the corruption seems pretty... Consistent, I guess*, I'd guess Nvidia's drivers are activating some internal feature suddenly that always causes the same issue in GZDoom and/or the Vulkan memory management library GZDoom uses. (And it probably only gets disabled again on restart)

* (e.g., those black spots in the top right corner are always there and always in that spot, textures other than the sky don't really seem to get corrupted, etc...)
User avatar
Nash
 
 
Posts: 17319
Joined: Mon Oct 27, 2003 12:07 am
Location: Kuala Lumpur, Malaysia

Re: Unclearable and constant SSAO lag

Post by Nash »

Is this "out of memory with Vulkan on nvidia" problem only unique to GZDoom, or can the same thing happen to other games that use Vulkan, like say, Doom 2016/Eternal?
dpJudas
 
 
Posts: 2890
Joined: Sat May 28, 2016 1:01 pm

Re: Unclearable and constant SSAO lag

Post by dpJudas »

You don't generally see this problem because newer games keep track on their total texture allocation budget and unloads textures if a new texture exceeds the budget. I.e. gzdoom could reload a 6x scaled texture at 3x or 1x if there isnt room for all of them. However, this requires improvements in how textures are handled.

In most games today the textures are always loaded in a low resolution (1x scale in gzd) and then the higher res are only streamed in for those close to the viewer. If you pay attention you can sometimes see this in the games where the texture is low res and then transforms into higher res as the texture loading thread catches up
User avatar
Graf Zahl
Lead GZDoom+Raze Developer
Lead GZDoom+Raze Developer
Posts: 48325
Joined: Sat Jul 19, 2003 10:19 am
Location: Germany

Re: Unclearable and constant SSAO lag

Post by Graf Zahl »

Modern games do not offer such a wasteful feature like a 6x upscaler. This was utter nonsense to add and certainly not my idea. The problem is that this is an option those clueless users will actually use, even on a 2GB graphics card - which gets essentially reduced to a 50 MB card with such high scaling factors - and we all know that some modern maps can use quite a high number of textures.
dpJudas
 
 
Posts: 2890
Joined: Sat May 28, 2016 1:01 pm

Re: Unclearable and constant SSAO lag

Post by dpJudas »

128x128 * 6 = 768x768. That's far less than the 4096x4096 textures used by modern games (which are fully PBR, so there's 4 or more of them per material). It is a large part of the reason games nowadays can be over 80 gigabytes in size.

Note: I'm not commenting on whether GZDoom should support something like this or not. I was replying to Nash asking why modern games don't run into the memory limit. And the answer to that is because of the way they stream textures.

Return to “Vulkan Renderer Bugs”