by Rachael » Sat Mar 26, 2022 2:19 pm
Unfortunately, we are unable to support external tools or renderers directly. If someone is willing to provide a patch that fixes this without interfering with GZDoom's normal operation we would be happy to accept it. But running GZDoom via any unsupported means - i.e. by wrappers, virtualization, rendering customization, or any other method which changes the behavior of the API on top of which GZDoom is run, is not something that we can control. We simply don't have the means to test every single software configuration and I feel that it would introduce an incredible amount of code bloat in order to account for every possible software-side API-modding situation.
We do have precedent for this already - several people in the past have attempted to run GZDoom with software such as Hypercam running into the background and have similarly run into problems. For that, just as we do in this particular instance - we had to recommend alternatives (such as OBS Studio) that would serve the same purpose.
The most likely cause of this is an unimplemented (or untested) feature on the modded API's side, which means you will have to talk to the vendor of vkBasalt for a solution.
(Yes I am aware that we talked in Discord about this and I converted one of the shaders you wanted to use to GZDoom - this unfortunately is the only method for using custom shaders that we can directly support)
As an anecdotal aside, for both Linux and Windows it is possible to hook into the window managers of both OS's and grab the images as they are being sent to the desktop and overlay a post-processed window on top of the application window. The input would be limited to the desktop's 24-bit color resolution, sure, but this method might be worthy of consideration for anyone wanting to develop future ReShade alternatives. (Not all applications render beyond 24-bit internally anyhow so it's not like that should be huge deal breaker) - This obviously would only work with the dynamic window managers, i.e. Windows Vista/7's DWM.exe implementation that was forced on with Win8 or later, or on Linux - XCompMgr or equivalents.