ZDoom

I think it's safe to say this has been fixed now.

So far so good, at least. No crashes, and the sky hasn't been discolored after running it a few separate times. It was an intermittent issue, so I can't say for 100% definite it's not a problem anymore, but I'm feeling confident after a few runs and a couple rebuilds, it hasn't occurred again.

Old C baggage. :?

Really puzzled why something like this isn't considered a compile error in C++

fixed that.

Thread 1 (Thread 0x7f28668ba440 (LWP 122804) "gzdoom"):
#0  0x00007f2867aeda67 in __GI___wait4 (pid=pid@entry=122840, stat_loc=stat_loc@entry=0x562bde157e74 <altstack+6804>, options=options@entry=0, usage=usage@entry=0x0) at ../sysdeps/unix/sysv/linux/wait4.c:30
No locals.
#1  0x00007f2867aed9e7 in __GI___waitpid (pid=pid@entry=122840, stat_loc=stat_loc@entry=0x562bde157e74 <altstack+6804>, options=options@entry=0) at waitpid.c:38
No locals.
#2  0x0000562bdd80b47c in crash_catcher (signum=11, siginfo=<optimized out>, context=<optimized out>) at /home/kitty/projects/gzdoom/src/common/platform/posix/sdl/crashcatcher.c:248
        status = 0
        dbg_pid = 122840
        fd = {49, 50}
#3  <signal handler called>
No locals.
#4  0x0000562bdd7d871b in VulkanImage::~VulkanImage (this=0x1100100000001, __in_chrg=<optimized out>) at /home/kitty/projects/gzdoom/src/common/rendering/vulkan/system/vk_objects.h:1063
No locals.
#5  std::default_delete<VulkanImage>::operator() (__ptr=0x1100100000001, this=<optimized out>) at /usr/include/c++/11/bits/unique_ptr.h:85
No locals.
#6  0x0000562bdd7d87f8 in std::default_delete<VulkanImage>::operator() (__ptr=<optimized out>, this=<optimized out>) at /usr/include/c++/11/bits/unique_ptr.h:79
No locals.
#7  std::unique_ptr<VulkanImage, std::default_delete<VulkanImage> >::~unique_ptr (this=0x562be5e9edb0, __in_chrg=<optimized out>) at /usr/include/c++/11/bits/unique_ptr.h:361
        __ptr = @0x562be5e9edb0: 0x1100100000001
#8  ImageBuilder::Create (this=<optimized out>, device=<optimized out>, allocatedBytes=<optimized out>) at /home/kitty/projects/gzdoom/src/common/rendering/vulkan/system/vk_builders.cpp:365
        image = 0x562be6809670
        allocation = 0x10101e0
        result = <optimized out>
        obj = std::unique_ptr<VulkanImage> = {get() = 0x1100100000001}
#9  0x0000000000000000 in ?? ()
No symbol table info available.

/home/kitty/projects/gzdoom/src/common/rendering/vulkan/system/vk_builders.cpp: In member function ‘GraphicsPipelineBuilder& GraphicsPipelineBuilder::BlendMode(const FRenderStyle&)’:
/home/kitty/projects/gzdoom/src/common/rendering/vulkan/system/vk_builders.cpp:263:18: warning: control reaches end of non-void function [-Wreturn-type]
  263 |         BlendMode((VkBlendOp)blendequation, (VkBlendFactor)srcblend, (VkBlendFactor)dstblend);
      |         ~~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

The PR for this was recently merged, however now it's crashing on load, after the window is created but before the title screen or anything comes up. It does [i]not[/i] crash in Debug mode (and everything including the sky looks correct), but RelWithDebInfo results in the attached crash log, Seems there's some stack smashing going on, given the backtrace:
[code]Thread 1 (Thread 0x7f28668ba440 (LWP 122804) "gzdoom"):
#0 0x00007f2867aeda67 in __GI___wait4 (pid=pid@entry=122840, stat_loc=stat_loc@entry=0x562bde157e74 <altstack+6804>, options=options@entry=0, usage=usage@entry=0x0) at ../sysdeps/unix/sysv/linux/wait4.c:30
No locals.
#1 0x00007f2867aed9e7 in __GI___waitpid (pid=pid@entry=122840, stat_loc=stat_loc@entry=0x562bde157e74 <altstack+6804>, options=options@entry=0) at waitpid.c:38
No locals.
#2 0x0000562bdd80b47c in crash_catcher (signum=11, siginfo=<optimized out>, context=<optimized out>) at /home/kitty/projects/gzdoom/src/common/platform/posix/sdl/crashcatcher.c:248
status = 0
dbg_pid = 122840
fd = {49, 50}
#3 <signal handler called>
No locals.
#4 0x0000562bdd7d871b in VulkanImage::~VulkanImage (this=0x1100100000001, __in_chrg=<optimized out>) at /home/kitty/projects/gzdoom/src/common/rendering/vulkan/system/vk_objects.h:1063
No locals.
#5 std::default_delete<VulkanImage>::operator() (__ptr=0x1100100000001, this=<optimized out>) at /usr/include/c++/11/bits/unique_ptr.h:85
No locals.
#6 0x0000562bdd7d87f8 in std::default_delete<VulkanImage>::operator() (__ptr=<optimized out>, this=<optimized out>) at /usr/include/c++/11/bits/unique_ptr.h:79
No locals.
#7 std::unique_ptr<VulkanImage, std::default_delete<VulkanImage> >::~unique_ptr (this=0x562be5e9edb0, __in_chrg=<optimized out>) at /usr/include/c++/11/bits/unique_ptr.h:361
__ptr = @0x562be5e9edb0: 0x1100100000001
#8 ImageBuilder::Create (this=<optimized out>, device=<optimized out>, allocatedBytes=<optimized out>) at /home/kitty/projects/gzdoom/src/common/rendering/vulkan/system/vk_builders.cpp:365
image = 0x562be6809670
allocation = 0x10101e0
result = <optimized out>
obj = std::unique_ptr<VulkanImage> = {get() = 0x1100100000001}
#9 0x0000000000000000 in ?? ()
No symbol table info available.[/code]
I've seen other random values for those final [c]__ptr[/c]s and [c]this[/c]s, including 0x1.

EDIT:
I think I found it:
[code]/home/kitty/projects/gzdoom/src/common/rendering/vulkan/system/vk_builders.cpp: In member function ‘GraphicsPipelineBuilder& GraphicsPipelineBuilder::BlendMode(const FRenderStyle&)’:
/home/kitty/projects/gzdoom/src/common/rendering/vulkan/system/vk_builders.cpp:263:18: warning: control reaches end of non-void function [-Wreturn-type]
263 | BlendMode((VkBlendOp)blendequation, (VkBlendFactor)srcblend, (VkBlendFactor)dstblend);
| ~~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~[/code]
The function is missing a return at the end. If I add [c]return *this;[/c], it stops crashing.

Moved back to Bugs since it's certainly not fixed yet, and is waiting on a PR.

Excellent. I'll PR that branch once I've disabled the ray query stuff on it (nvidia crashes with that on a driver worker thread with a null pointer exception - thanks nvidia!). It contains a much newer version of the backend.

Works fine on that branch.

Hmm that sure looks weird! How does it look like on [url=https://github.com/dpjudas/gzdoom/tree/rtx]this branch[/url]?

Well... it's certainly different now but seems that's not quite the fix.
[attachment=0]Screenshot_Doom_20220613_235542.png[/attachment]

Thank you

And here's the [url=https://github.com/coelckers/gzdoom/pull/1634]pull request[/url] that fixes it.

I finally figured out what is going on here.

The lightmap branch introduced a texture for the light map, but if there is no actual lightmap the vulkan backend creates a dummy lightmap texture as the pipeline needs one bound at all times regardless of whether the texture is actually used by the shader or not.

This dummy texture is created with undefined contents. On Nvidia's driver this creates a 1x1 texture with a black pixel. On AMD/Mesa it creates whatever is in memory. That's why the bug doesn't manifest itself on my computer.

Now for the actual bug in the code:

The sky is drawn using the skydome code. It only provides vertex attributes for the SIMPLE vertex shader (aPosition, aTexCoord, aColor). But the actual shader used is not a SIMPLE shader. Thus half the vertex attributes aren't assigned at all! Yay for using every exotic/bad idea feature in OpenGL.

Anyway, this results in main.fp's check for lightmaps being active to fail and thus it samples from the lightmap texture. Due to NV always clearing the texture that didn't have any visual effect at all on my computer, but on AMD hardware it does.

Time to implement glVertexAttrib in this OpenGL emulator...

The sky indeed is restored with that line commented out.

Edit: Also it seems the purple vs blacked out is entirely luck of the draw and they are the same bug.

Edit 2: And for whatever it's worth, the shader seems to be fine under Zink with RADV. Mesa 21.1's lavapipe also seems to be fine, so it could still be a RADV bug but it is interesting that Zink circumvents it if it is.

Edit 3: The bug appears in both Mesa's RADV and AMD's AMDVLK.