Black screen on macOS with NVIDIA graphics

[_mental_]

I spent a lot of time in futile attempts to fix the black screen with NVIDIA graphics, GeForce 750M.

Tried with MoltenVK and Vulkan Portability Implementation, with Cocoa and SDL backends to no avail.
Comparison of Xcode GPU frame captures didn't reveal any reason while target textures are pure black when NVIDIA hardware is used.
Simplifications of shaders and changes to output formats didn't change anything as well.
Vulkan SDK versions were 1.1.101 and 1.1.106 downloaded from LunarG website.

Validation layers produce only one warning that is also present with integrated Intel Iris Pro graphics.

Code: Select all

[vulkan error] Queue 0x600002c1fd10 is signaling semaphore 0x5 that was previously signaled by queue 0x600002c1fd10 but has not since been waited on by any queue.

---

Code: Select all

QueueSubmit::execute(this=0x00007ffeefbfc498, device=0x00000001047b1e00, queue=0x0000600002c1fd10, fence=0x000060000006c740) at vk_builders.h:1236:20
VulkanFrameBuffer::FlushCommands(this=0x0000000103ec77a0, commands=0x00007ffeefbfc640, count=1, finish=true) at vk_framebuffer.cpp:221:9
VulkanFrameBuffer::FlushCommands(this=0x0000000103ec77a0, finish=true) at vk_framebuffer.cpp:246:3
VulkanFrameBuffer::WaitForCommands(this=0x0000000103ec77a0, finish=true) at vk_framebuffer.cpp:264:2
VulkanFrameBuffer::Update(this=0x0000000103ec77a0) at vk_framebuffer.cpp:182:2
DFrameBuffer::End2DAndUpdate(this=0x0000000103ec77a0) at v_video.h:462:3
D_Display() at d_main.cpp:899:11

However, with vk_debug CVAR was set, the screen was black too even with Intel GPU.

Outside of the validation issue, I would like to ask about anything else I can check that may help to fix this problem.
Also, it will nice if someone will test vulkan2 branch on Mac with AMD graphics.

[dpJudas]

Okay, I fixed the validation error.

About the black screen, we need to figure out if its the rendering or the presentation that fails. You can call CopyScreenToBuffer in VulkanFrameBuffer::Update, just before it calls WaitForCommands. That should copy the contents of what was rendered to an image. Something like this:

Code: Select all

void VulkanFrameBuffer::Update()
{
...

	int width = GetWidth();
	int height = GetHeight();
	int numpixels = width * height;
	uint8_t * scr = (uint8_t *)M_Malloc(numpixels * 3);
	CopyScreenToBuffer(width, height, scr);
	// Save the image to disk here
	M_Free(scr);

	WaitForCommands(true);

	Super::Update();
}

The shader that controls the final copy to the swap chain image is Present.fp - might also be worth a try to make it output a solid color or its texture coordinates.

[_mental_]

The picture obtained with code added to VulkanFrameBuffer::Update() contains only black pixels.
Replacing main() function of present.fp with the following snippet had no effect at all.

Code: Select all

void main()
{
	FragColor = vec4(.5, .5, .5, 1.);
}

[dpJudas]

Hmm, weird. Almost as if it doesn't render anything at all.

[dpJudas]

Try replace screenquad.vp with this shader:

Code: Select all

layout(location = 0) out vec2 texCoord;

vec2 positions[6] = vec2[](
    vec2(-1.0, -1.0),
    vec2( 1.0, -1.0),
    vec2(-1.0,  1.0),
    vec2(-1.0,  1.0),
    vec2( 1.0, -1.0),
    vec2( 1.0,  1.0)
);

void main()
{
    vec4 pos = vec4(positions[gl_VertexIndex%6], 0.0, 1.0);
    gl_Position = pos;
    texCoord = pos.xy * 0.5 + 0.5;
}

It creates the vertices needed for the screen quad by itself without relying on the vertex buffer. Use that with your modified present.fp.

Edit: by the way, do I need to install something extra on my mac to get it to use vulkan?

[_mental_]

Now at least something is visible. Without solid color in fragment program whole screen is still black.

by the way, do I need to install something extra on my mac to get it to use vulkan?

You need only libvulkan.dylib or libvulkan.dylib.1 or libMoltenVK.dylib in the loader path.
If there is no system-wide installation of Vulkan SDK, .dylib can be copied next to executable, to gzdoom.app/Contents/MacOS.

To enable validation layers, setup is a bit trickier.
There is macOS directory inside LunarG SDK distribution. Unpack it somewhere, let's say to /Users/me/vulkan folder.
Three environment variables should be set like these

Code: Select all

DYLD_LIBRARY_PATH=/Users/me/vulkan/lib
VK_LAYER_PATH=/Users/me/vulkan/etc/vulkan/explicit_layer.d
VK_ICD_FILENAMES=/Users/me/vulkan/etc/vulkan/icd.d/MoltenVK_icd.json

They can be copy-pasted into Xcode for the current scheme.

[dpJudas]

Now at least something is visible. Without solid color in fragment program whole screen is still black.

Excellent! Now we are getting somewhere. The presentation is working, but it is failing to render anything due vertex buffer problems or maybe broken winding order setup.

I can see on the code that the culling mode is none, which rules out the screen quad getting culled. That must mean it is a problem with the vertex buffer one way or another. Big question is what. This one is going to be a little tricky to debug unless you have some debugger tool on Mac that can display the contents of the vertex buffer used as input to the present shader. Seems RenderDoc is out of the question.

[_mental_]

It's something with vertex buffer indeed. This is how the last DIP was renderer with Intel GPU.

This is with NVIDIA. There were two warnings about degenerated triangles.

---

Code: Select all

Row	Offset	PositionInProjection			UV	
0	0x0	 0.000000	 0.000000	 0.000000	 0.000000	 0.000000
1	0x14	 1.000000	 0.000000	 0.000000	 0.000000	 0.000000
2	0x28	 2.000000	 0.000000	 0.000000	 0.000000	 0.000000
3	0x3C	 3.000000	 0.000000	 0.000000	 0.000000	 0.000000
4	0x50	 0.000000	 0.000000	 0.000000	 0.000000	 0.000000
5	0x64	 0.000000	 720.000000	 0.000000	 0.000000	 1.000000
6	0x78	 1152.000000	 0.000000	 0.000000	 1.000000	 0.000000
7	0x8C	 1152.000000	 720.000000	 0.000000	 1.000000	 1.000000
8	0xA0	-1.000000	-1.000000	 0.000000	 0.000000	 0.000000
9	0xB4	-1.000000	 1.000000	 0.000000	 0.000000	 1.000000
10	0xC8	 1.000000	-1.000000	 0.000000	 1.000000	 0.000000
11	0xDC	 1.000000	 1.000000	 0.000000	 1.000000	 1.000000
12	0xF0	-32767.000000	 32767.000000	-32767.000000	 0.000000	 0.000000
13	0x104	-32767.000000	 32767.000000	 32767.000000	 0.000000	 0.000000
14	0x118	 32767.000000	 32767.000000	 32767.000000	 0.000000	 0.000000
15	0x12C	 32767.000000	 32767.000000	-32767.000000	 0.000000	 0.000000
16	0x140	-32767.000000	-32767.000000	-32767.000000	 0.000000	 0.000000
17	0x154	-32767.000000	-32767.000000	 32767.000000	 0.000000	 0.000000
18	0x168	 32767.000000	-32767.000000	 32767.000000	 0.000000	 0.000000
19	0x17C	 32767.000000	-32767.000000	-32767.000000	 0.000000	 0.000000
20	0x190	 0.000000	 0.000000	 0.000000	 0.000000	 0.000000
...

Content of vertex buffer with NVIDIA GPU are all zeroes.

At the moment I have no idea how to track the source of wrong data.

[dpJudas]

This particular vertex buffer is being created and filled in FFlatVertexBuffer::FFlatVertexBuffer. I'm not sure how anything can fail here since it does a memcpy in FFlatVertexBuffer::Copy with a destination pointer returned by Vulkan (technically, by our vma memory allocation library). If the mapping operation had returned a null pointer it would have crashed right there in the memcpy.

[_mental_]

Flat vertex buffer data are copied correctly. It seems like driver and/or graphics hardware doesn't receive them.
Maybe it's related to integrated vs. discrete GPU.

I tested Vulkan samples and vkQuake 2 with NVIDIA graphics and they work fine.

[dpJudas]

Does any of those samples (or vkQuake 2) use persistently mapped vertex buffers? I don't have any theory for why it doesn't receive them.

As far as I understand the vma usage flags the buffer should have VK_MEMORY_PROPERTY_HOST_COHERENT_BIT. I guess you could try call vkFlushMappedMemoryRanges or vkInvalidateMappedMemoryRanges and see if it makes any difference. Or at least figure out what the current flags are for the alloacted vertex buffer.

[_mental_]

Does any of those samples (or vkQuake 2) use persistently mapped vertex buffers? I don't have any theory for why it doesn't receive them.

I didn't check source codes to be honest. But even if I did, Vulkan API is rather complicated, so there is a high chance to miss important details for an inexperienced user.
What is the exact thing I need to look for?

As far as I understand the vma usage flags the buffer should have VK_MEMORY_PROPERTY_HOST_COHERENT_BIT. I guess you could try call vkFlushMappedMemoryRanges or vkInvalidateMappedMemoryRanges and see if it makes any difference. Or at least figure out what the current flags are for the alloacted vertex buffer.

Unfortunately, this tells me mostly nothing. To be able to make some tests, I need more details about changes I should do.
If you have a Mac that supports Metal, probably it will easier to check yourself than to explain what to do.

[dpJudas]

I unfortunately do not have a Mac with NVidia graphics. I'll try create a debugging branch where it outputs some of the info I'm curious about - mostly the properties for the memory vma allocated for the buffer, so we can see what guarantees vulkan gave up. Probably can also throw in a few calls to vkFlushMappedMemoryRanges and vkInvalidateMappedMemoryRanges to see if that somehow makes the memory visble for the GPU.

I can't say exactly when I'll get around to this though. Some personal things came up that I have to deal with first. I'll reply back once I have something you can use for further testing.

[dpJudas]

I just discovered (and fixed with this commit) that the VMA_MEMORY_USAGE_CPU_TO_GPU mode used by our buffers does not require the memory to be coherent. It is possible that the black screen happened because it picked a memory type that wasn't coherent.

Please try again with latest master. Hopefully it now works or it will throw an error about not finding any suitable memory.

[_mental_]

Thanks, this commit fixed the problem.