I am making this post to point to some potential breaking changes I had to make in order to fix this feature.
Recently I decided to fix the mess that is 'RenderStyle "Shaded"' (a.k.a. alpha textures)
For those who do not know, this is the style which gets used for blood and scorch mark decals.
A long standing problem with this style has been its poor and inconsistent implementation. Depending on what type of texture was passed and what renderer was active it either worked or did not - in an entirely unintuitive manner. The original software renderer's implementation never went beyond the barest necessities to make shaded decals work, and this never got changed when the feature was extended to a true render style.
The OpenGL hardware renderer has always been able to handle this properly for grayscale textures and produced ok results for other non-paletted textures. It was merely ok for those, because for true color textures the red channel was mapped to alpha, which not only is not the brightest one (that'd be green) and it'd produce bad results for anything with low red content.
The same hasn't been true for the software renderer. This style only worked there correctly when using either IMGZ or grayscale PNGs that got tagged with a custom 'alPh' chunk.
It got even worse with paletted formats like Doom patches. In this case it completely ignored the colors and treated the image's content as a grayscale ramp, but did not consider that the texture manager remaps color 0 to reserve that index for transparency.
Needless to say, with something this inconsistent and seemingly broken by design, the feature was basically reduced to its original use case, i.e. shaded decals and crosshairs and never got beyond that.
I always stayed away from fixing this because I knew that a) it would be a lot of work and b) might have an impact on older mods that depend on the broken semantics here.
Fixing this to behave in a sane fashion would inevitably break any old mod that uses a Doom patch for a decal, unless some countermeasures were implemented.
I finally did some checks of available mods to see what I am up against by checking the entire /idgames content for use of undefined side effects of the old implementation.
The result of this was: roughly 140 patch textures were used for decals over the years, the most recent ones dating back to 2011 and two IMGZ textures that use this format for a color texture.
The most important affected mod is Beautiful Doom whose decals got reused by several other mods, making proper treatment for these a necessity.
The major part of the rest of these 140 textures comes from two mods (njol.wad, njbgpabh.pk3) which merely reused some old, already existing data. Aside from these three mods and the reused Beautiful Doom content, there were only 5-10 affected graphics.
With this amount it was doable to implement a checklist for the affected textures and then fix all those design problems here, which is what I did over the last week.
With the rewritten code, render style "shaded" will now always mean 'take the grayscale value of each pixel and use it as alpha with the current stencil/fill color'.
This implies some important changes to how textures behave:
1) IMGZ will from now on be treated as a grayscale-only format and be considered deprecated. Please discontinue using this format if you still use this for anything else but decals and crosshairs!
2) Doom patches will always be treated as a paletted image, even when being used as an alpha texture. This means that any mod still using such patches (aside from the known ones from Beautiful Doom) will display broken shaded decals! Again, please discontinue using patches for shaded decals, if you still do this.
3) It is no longer necessary to tag alpha PNGs with an 'alPh' chunk. In fact, this will not be checked for anymore by the texture manager.
4) If an image works as alpha texture in one renderer it will now work in all. All those crosshairs from Project Brutality, for example, that were broken with OpenGL 2.0 will now be displayed correctly. So no more trouble with badly prepared graphics that depend on how the renderers implement the feature
So how to deal with this:
1) If you make decals, use a graphics format that gives you full control over the colors like PNG - although JPG may also be an alternative for larger decals now.
2) Known images that use a format incorrectly will be flagged through a list of MD5 hashes to switch from color to grayscale (for patches) or from grayscale to color (for IMGZ.)
3) If you find some old mods with broken decals, please make a bug report pointing to that file. They will be added to the internal exception list then.
4) Note that 3) only applies for released mods as of March 2018! To consider adding something to the exception list, there must be some public link to the file. We cannot and will not extend this list beyond what is needed for backwards compatibility. For WIP you will have to switch to PNGs.
While some people may cry foul over some old feature being altered in a way that might affect a few mods let's not forget one thing:
Due to the messy and inconsistent implementation I have found far, far more decals in /idgames that were plain and simply broken because modders did not properly adhere to the very specific limitations. So by removing those limitations it can be expected that in future mods there will be less bad definitions.
In the end the choice was to do the lesser of two evils. Either keep the feature broken and mostly useless forever or fix it and deal with some potential issues in a very small number of old mods. Most mods using decals were made long after PNG became the image format of choice for ZDoom modding and will not be affected. In fact, a few mods that did not use the feature properly may now even work better than before.