RenderStyle "Shaded" - or, fixing a totally broken feature

[Graf Zahl]

I am making this post to point to some potential breaking changes I had to make in order to fix this feature.

Recently I decided to fix the mess that is 'RenderStyle "Shaded"' (a.k.a. alpha textures)
For those who do not know, this is the style which gets used for blood and scorch mark decals.

A long standing problem with this style has been its poor and inconsistent implementation. Depending on what type of texture was passed and what renderer was active it either worked or did not - in an entirely unintuitive manner. The original software renderer's implementation never went beyond the barest necessities to make shaded decals work, and this never got changed when the feature was extended to a true render style.

The OpenGL hardware renderer has always been able to handle this properly for grayscale textures and produced ok results for other non-paletted textures. It was merely ok for those, because for true color textures the red channel was mapped to alpha, which not only is not the brightest one (that'd be green) and it'd produce bad results for anything with low red content.
The same hasn't been true for the software renderer. This style only worked there correctly when using either IMGZ or grayscale PNGs that got tagged with a custom 'alPh' chunk.
It got even worse with paletted formats like Doom patches. In this case it completely ignored the colors and treated the image's content as a grayscale ramp, but did not consider that the texture manager remaps color 0 to reserve that index for transparency.

Needless to say, with something this inconsistent and seemingly broken by design, the feature was basically reduced to its original use case, i.e. shaded decals and crosshairs and never got beyond that.

I always stayed away from fixing this because I knew that a) it would be a lot of work and b) might have an impact on older mods that depend on the broken semantics here.
Fixing this to behave in a sane fashion would inevitably break any old mod that uses a Doom patch for a decal, unless some countermeasures were implemented.

I finally did some checks of available mods to see what I am up against by checking the entire /idgames content for use of undefined side effects of the old implementation.
The result of this was: roughly 140 patch textures were used for decals over the years, the most recent ones dating back to 2011 and two IMGZ textures that use this format for a color texture.
The most important affected mod is Beautiful Doom whose decals got reused by several other mods, making proper treatment for these a necessity.
The major part of the rest of these 140 textures comes from two mods (njol.wad, njbgpabh.pk3) which merely reused some old, already existing data. Aside from these three mods and the reused Beautiful Doom content, there were only 5-10 affected graphics.

With this amount it was doable to implement a checklist for the affected textures and then fix all those design problems here, which is what I did over the last week.
With the rewritten code, render style "shaded" will now always mean 'take the grayscale value of each pixel and use it as alpha with the current stencil/fill color'.

This implies some important changes to how textures behave:
1) IMGZ will from now on be treated as a grayscale-only format and be considered deprecated. Please discontinue using this format if you still use this for anything else but decals and crosshairs!
2) Doom patches will always be treated as a paletted image, even when being used as an alpha texture. This means that any mod still using such patches (aside from the known ones from Beautiful Doom) will display broken shaded decals! Again, please discontinue using patches for shaded decals, if you still do this.
3) It is no longer necessary to tag alpha PNGs with an 'alPh' chunk. In fact, this will not be checked for anymore by the texture manager.
4) If an image works as alpha texture in one renderer it will now work in all. All those crosshairs from Project Brutality, for example, that were broken with OpenGL 2.0 will now be displayed correctly. So no more trouble with badly prepared graphics that depend on how the renderers implement the feature

So how to deal with this:

1) If you make decals, use a graphics format that gives you full control over the colors like PNG - although JPG may also be an alternative for larger decals now.
2) Known images that use a format incorrectly will be flagged through a list of MD5 hashes to switch from color to grayscale (for patches) or from grayscale to color (for IMGZ.)
3) If you find some old mods with broken decals, please make a bug report pointing to that file. They will be added to the internal exception list then.
4) Note that 3) only applies for released mods as of March 2018! To consider adding something to the exception list, there must be some public link to the file. We cannot and will not extend this list beyond what is needed for backwards compatibility. For WIP you will have to switch to PNGs.

While some people may cry foul over some old feature being altered in a way that might affect a few mods let's not forget one thing:
Due to the messy and inconsistent implementation I have found far, far more decals in /idgames that were plain and simply broken because modders did not properly adhere to the very specific limitations. So by removing those limitations it can be expected that in future mods there will be less bad definitions.

In the end the choice was to do the lesser of two evils. Either keep the feature broken and mostly useless forever or fix it and deal with some potential issues in a very small number of old mods. Most mods using decals were made long after PNG became the image format of choice for ZDoom modding and will not be affected. In fact, a few mods that did not use the feature properly may now even work better than before.

[Kinsie]

Thanks for the heads-up. I just checked my mods in the latest devbuild (since they use actors with the Shaded renderstyle for certain visual effects) and nothing seemed to be broken.

[Matt]

Would stuff like this still be acceptable?

---

Code: Select all • Expand view

decal PlasmaShock1
{
 pic PLSPA0
 fullbright
 add 0.6
 x-scale 0.2
 y-scale 0.2
 randomflipx
 randomflipy
 lowerdecal PlasmaScorchLower
 animator GoAway
}
decal PlasmaShock2
{
 pic PLSPB0
 fullbright
 add 0.6
 x-scale 0.2
 y-scale 0.2
 randomflipx
 randomflipy
 lowerdecal PlasmaScorchLower1
 animator GoAway
}
decal PlasmaShock3
{
 pic PLSPC0
 fullbright
 add 0.6
 x-scale 0.2
 y-scale 0.2
 randomflipx
 randomflipy
 lowerdecal PlasmaScorchLower1
 animator GoAway
}
decalgroup PlasmaShock
{
 PlasmaShock1 1
 PlasmaShock2 1
 PlasmaShock3 1
}
decal GooScorch
{
 pic BAL7E0
 x-scale 0.7
 y-scale 0.7
 add 0.6
 lowerdecal Scorch
 animator GoAway2
} 

[Graf Zahl]

None of those use the shaded render style, so they were never affected by the problems.

[Matt]

Thanks! I've always been super confused as to what shaded did or what it was useful for, so I never paid attention to it and wanted to make sure given the "if you make decals" that I wasn't using it unknowingly.

[Graf Zahl]

In that case, good news for you: You DON'T have to think about it anymore because it no longer needs any special treatment.

[Nash]

Making ZDoom-friendly blood decals that look exactly like the stock blood decals has always been a hit-and-miss experience for me.

Good to know now I can just provide good ol' greyscale PNGs for my blood splat replacements. Thanks for fixing this!

[Enjay]

I really can't remember with any degree of certainty why my old mods use paletted graphics for decals (etc) but I *think* it's because the lumps date back to a time when ZDoom first (or at least relatively recently) introduced decals. I didn't know the format but I *think* DeePsea would extract them as Doom palette BMPs using their grayscale palette location as the position in the Doom palette. So I then made my own graphics trying to use that kind of placement somehow. (Not sure how, probably using some kind of palette mapping in Paint Shop Pro.) Then I stuck them in a WAD as Doom format patches... I think.

It was a decision made from ignorance, not understanding the format and working with what seemed to be right/working according to the tools that I had to hand. Who knew that so many years on (maybe even as much 2 decades?) it was going to cause a problem.

Great to hear that the system is now properly implemented with standard graphics.

And to answer a question posed in the original Dev thread, yeah, my old Aspects files probably are full of out-dated/obsolete format data types but, as far as I'm concerned, the project really isn't a thing any more (hasn't been for a very long time). I do still use the resources (most of which are already public) to use as an editing resource when I want to mess around with personal stuff every now and again though. So, if I was trying to use something from those resource files and found that the resource no longer worked, I might post to check if it was meant to work or not but I would also usually be in a position to create a new/correct format version for anything that was needed for either a personal project or a public release.

For what it's worth, Ed The Bat and I have (very slowly) been working on an updated Burghead so something like the problems raised with this mod could/should get fixed if we ever get near completion and release.

[Matt]

updated Burghead

Best [Doom-related] news I've heard all day!