[1.0.0] Visual Artifacting in Vulkan mode

[Graf Zahl]

Yeah, they invented this shit because they designed defective hardware. But it has always been the same with them on OpenGL. Even if the spec was broken, they were anal about ensuring that their implementation replicates the brokenness which only caused grief among developers.
And you can't tell me that this convoluted way of uploading textures is more efficient than letting the driver handle all the gory details internally. It only forces the same overcomplicated setup on hardware that doesn't need it.

TBH, if it wasn't for Linux or Mac I'd have tossed out Vulkan long ago and replaced it with a D3D11 backend. That API does not torture the poor developers with all this shit and still seems to work fine for projects that don't push the GPU to the absolute limit.

[Rachael]

From what I understand, with the availability of DXVK I think some developers on Linux have already started saying screw it to both OpenGL and Vulkan and going straight for D3D11, and using the wrapper library to support Linux.

In terms of Windows, supporting D3D11 is a good idea anyway, because even if your video card does not support the API, Windows will fill in the gaps with some sort of emulation (Win7 and later only). Sure it's slow as hell, but at least it still works.

[Graf Zahl]

From what I understand, with the availability of DXVK I think some developers on Linux have already started saying screw it to both OpenGL and Vulkan and going straight for D3D11, and using the wrapper library to support Linux.

Fully understandable - Vulkan is clearly good for professional 3D engines made by people who think in these terms, but it's an utter horror story for normal programmers who just want to get things done. The irony for GZDoom and Raze here is that it's the rigid and fault-intolerant design here that has kept me from actually doing some render flow optimizations, making the whole thing essentially falling flat on its face where it really should have mattered.

In terms of Windows, supporting D3D11 is a good idea anyway, because even if your video card does not support the API, Windows will fill in the gaps with some sort of emulation (Win7 and later only). Sure it's slow as hell, but at least it still works.

With D3D11 we'd surely have better coverage at the low end of the spectrum while essentially being future proof. On Windows it's not going away for a long time and these wrapper libraries are also guaranteeing that it will stay around.
Let's see what my free time schedule has to say over the next months.

[Rachael]

I kept wanting to create a D3D11 backend for GZDoom just to get *some* experience in graphics API programming but kept getting discouraged because of how quickly it is likely to become deprecated and useless.

[Graf Zahl]

because of how quickly it is likely to become deprecated and useless.

I don't think it will ever become deprecated. After the initial enthusiasm about those new to-the-metal APIs I think reality has sunk in by now with many developers realizing that they aren't the universal savior for every use case.

[dpJudas]

Those errors are saying that some of the images are in the wrong image layout required for the operation.

To copy from a buffer to an image the format must be in VK_IMAGE_LAYOUT_TRANSFER_DST_OPTIMAL. If the image is being copied from one image to another it must be in VK_IMAGE_LAYOUT_TRANSFER_SRC_OPTIMAL. If the image is being used by a sampler it must be in VK_IMAGE_LAYOUT_SAMPLER_OPTIMAL. If the image is used as a render buffer it must be VK_IMAGE_LAYOUT_COLOR_ATTACHMENT_OPTIMAL.

For a camera texture the flow is: VK_IMAGE_LAYOUT_UNDEFINED -> VK_IMAGE_LAYOUT_COLOR_ATTACHMENT_OPTIMAL -> render scene -> VK_IMAGE_LAYOUT_SAMPLER_OPTIMAL -> bind to sampler -> VK_IMAGE_LAYOUT_COLOR_ATTACHMENT_OPTIMAL -> render scene -> VK_IMAGE_LAYOUT_SAMPLER_OPTIMAL -> bind to sampler...

For a normal texture with no mipmaps the flow is: VK_IMAGE_LAYOUT_UNDEFINED -> VK_IMAGE_LAYOUT_TRANSFER_DST_OPTIMAL -> copy from buffer to image -> VK_IMAGE_LAYOUT_SAMPLER_OPTIMAL -> bind to sampler.

For one with mipmaps the flow becomes the wonderful version of: VK_IMAGE_LAYOUT_UNDEFINED -> VK_IMAGE_LAYOUT_TRANSFER_DST_OPTIMAL -> copy from buffer to image -> VK_IMAGE_LAYOUT_TRANSFER_SRC_OPTIMAL -> blit scaled to next mipmap -> VK_IMAGE_LAYOUT_SAMPLER_OPTIMAL. The next mipmap needs to be in VK_IMAGE_LAYOUT_TRANSFER_DST_OPTIMAL when the blit happens, then converted to VK_IMAGE_LAYOUT_TRANSFER_SRC_OPTIMAL before it can be used to copy to the next mipmap level, then VK_IMAGE_LAYOUT_SAMPLER_OPTIMAL after the copy is done.

The output log reports many of the images are in the incorrect layout. That includes both scene buffer images and hardware texture images.

Does it also generate these errors with GZDoom?

[InsanityBringer]

On my machine, some validation errors are thrown in GZDoom, but none are related to texture transitions. Log attached, if you're curious.

[Graf Zahl]

I get 3 such errors when the engine starts up but none within the level itself.

Code: Select all

[vulkan error] vkCmdDrawIndexed(): Cannot use image 0xd5 (layer=0 mip=0) with specific layout VK_IMAGE_LAYOUT_UNDEFINED that doesn't match the actual current layout VK_IMAGE_LAYOUT_SHADER_READ_ONLY_OPTIMAL. The Vulkan spec states: imageLayout must match the actual VkImageLayout of each subresource accessible from imageView at the time this descriptor is accessed as defined by the image layout matching rules (https://www.khronos.org/registry/vulkan/specs/1.1-extensions/html/vkspec.html#VUID-VkDescriptorImageInfo-imageLayout-00344)
Called from VulkanDevice::DebugCallback at C:\Programming\Doom-Dev\GZDoom\src\common\rendering\vulkan\system\vk_device.cpp, line 405


[vulkan error] Descriptor set VkHardwareTexture.mDescriptorSets bound as set #1 encountered the following validation error at vkCmdDrawIndexed() time: Image layout specified at vkUpdateDescriptorSet* or vkCmdPushDescriptorSet* time doesn't match actual image layout at time descriptor is used. See previous error callback for specific details.
Called from VulkanDevice::DebugCallback at C:\Programming\Doom-Dev\GZDoom\src\common\rendering\vulkan\system\vk_device.cpp, line 405


[vulkan error] vkCmdDrawIndexed(): Cannot use image 0xd5 (layer=0 mip=0) with specific layout VK_IMAGE_LAYOUT_SHADER_READ_ONLY_OPTIMAL that doesn't match the actual current layout VK_IMAGE_LAYOUT_UNDEFINED. The Vulkan spec states: imageLayout must match the actual VkImageLayout of each subresource accessible from imageView at the time this descriptor is accessed as defined by the image layout matching rules (https://www.khronos.org/registry/vulkan/specs/1.1-extensions/html/vkspec.html#VUID-VkDescriptorImageInfo-imageLayout-00344)

But reading this stuff I seriously have to question the designers' sanity. How is this ever supposed to work without tripping all over it?

[Graf Zahl]

Another question: Can we retrieve the problem textures by name? If we could print that in the list we may have a few more clues what's up here.

[dpJudas]

Yes and no. The SetDebugName function that I have on most of the vulkan objects tells the debug layer what we want to call it (right now it just calls them all VkHardwareTexture.Image if I remember correctly), but the debug layer doesn't write the name in all cases.

The worst part about all of this isn't so much the image transitions have to be in order, but the way the debug layer makes it a nightmare to figure out what it is talking about.

By the way: There is a vk_debug_breakpoint CVAR that will call the debugger whenever the debug layer reports an error. That might help give you a stack trace and the variables in play.

[Graf Zahl]

The worst part about all of this isn't so much the image transitions have to be in order, but the way the debug layer makes it a nightmare to figure out what it is talking about.

Indeed, that's also my main issue here. These messages are often a horrible gobbledygook.

[Rachael]

I now have a RX 550, as soon as I update the drivers on it I will test this issue and see what I can do for it.

EDIT: And bug is definitely confirmed. I will take a look at this later when I am more awake.

[Rachael]

It appears the mystery behind this one is solved. I pulled some changes from GZDoom's Vulkan backend that somehow didn't make it to Raze in its corresponding backend update. Vulkan should now work for both the new renderer and for polymost in Raze on ATI's.

(For reference, I believe the issue was a mismatch between the image format for textures being placed on the buffer, and the buffer's own image format)