ZDoom

You really needed long time to figure out that "out of vram" is the issue

I figured it out directly and without all these debugging tools.
Just have MSI Afterburner Overlay running and you can see how the game freezes as soon as it reaches the maximum VRAM you have.

It either freezes completely or the game collapses down to below 5 fps.

With OpenGL it works fine.

My card is a RTX 2080 Ti.

Hey since you're so smart, can't you file a pull request with the fix? Surely it shouldn't be a problem.

I wish I could
_
Wasnt meant as offense. I was just a bit surprised that you didnt use overlay tools first which would've saved you so much time finding the issue

I cant write code though. Would be nice if you can somehow fix it

The issue is a little bit more complex than simply out of memory. The nvidia driver provides the application with some memory heaps and then cannot handle if they are fully used as adverticed.

_mental_ posted earlier test results showing that Intel works correctly where it produces an out of memory error. And Marisa showed that the Nvidia driver advertices larger heaps than there's video memory. The issue could very well be a Nvidia driver bug.

But how do other modern games handle it? Do they have a handler like OpenGL has, that it copies the too much VRAM over to System RAM, while the vulkan library in GZDoom currently doesnt?
So even if the nvidia driver sends too much vram, couldnt the too much data just be copied further to system ram then?

De-M-oN - just stop while you're ahead, please...

This is very pedantic and insulting.

...
Moving on.
With Graf's overhaul of the exit handling, perhaps now it'll throw the error instead of freezing. I'll give it a try soon, see if this has improved any at the very least. I know it most likely still won't help but it's still worth a shot.

I doubt it, especially if it's the driver causing issues. As far as I know, driver issues are something that the kernel has to handle, rather than the application. The driver may be able to forward the error to the application, but then we have this chicken-and-egg problem where we are right back where we started here in the first place.

Any allocator should return null if it cannot allocate the requested storage. This one does not, it just misbehaves. This needs to be fixed on NVidia's side.
The exit handling has nothing to do with it, that was done because some parts of code cannot be shut down from an exit handler, because the order of execution of them is problematic when DLLs are involved. And the DLLs for the sound backend had problems on some systems. Another reason was that the entire startup procedure could not be cleanly controlled because there was no good way to partially shut it down.

@De-Mo-oN: Video memory is a shared resource. Even if you explicitly check how much video memory is available you still run into the possibility that another application used a significant portion of it. The upper limit on the memory heaps in Vulkan is simply to prevent each individual application from using an excessive amount, and to provide hints how much memory is available in total. It is still expected that the vulkan display driver handles any required paging to system memory if the video memory ends up over committed. At least that was my impression when I read the spec (although if someone has seen a section in the spec that states otherwise please provide a link).

As for how other games do it - well obviously they limit how much video memory they allocate. But you need to understand this thing is really about two separate things: proper error handling for out of memory conditions, and better resource management in GZDoom. This thread is about the former, not the latter. So "use less memory, duh" is not really the core of the discussion.