[3.7.0] Models with Stencil renderstyle have no textures

[Korshun]

We have updated The Forestale from GZDoom 3.4.1 to GZDoom 3.7.2 and some models render without textures:

[imgur]https://imgur.com/a/74E5L7j[/imgur] (the imgur tag doesn't work for some reason)

They only render without textures if they are opaque. Turns out, it is caused by the Stencil renderstyle. We use the it to give models a colored tint. The bug got introduced somewhere between GZDoom 3.6.0 and 3.7.0.

[Graf Zahl]

The Stencil render style is SUPPOSED to render without texture. If you used it to tint something textured, that was actually incorrect use of it.

[Cherno]

I think you need to use AddStencil for the way older version of GZDoom used to render the Stencil renderstyle.

[Graf Zahl]

Render style stencil is supposed to render the non-transparent parts of a texture in a solid color.
There has been a bug that this wasn't true in all situations, that got fixed a while back.

[Korshun]

AddStencil still renders without a texture, just makes the model have additive translucency. What render style should be used to colorize models without losing the texture?

[Rachael]

There is none currently supported that will do that. A new renderstyle would have to be created, and that is currently quite an involved process considering how deep the renderstyle code goes into the engine, currently.

[Graf Zahl]

The big problem with a new render style is that this wouldn't just need one render style, but several, considering that it can be combined with various translucency modes.
The legacy render style enum is really showing its limits here, to do this without causing too many problems, the flag based render style needs to be exposed to editing first, but that's something I always kept off because there's too much of a chance to create a bad style that doesn't work.

[Korshun]

If so, could you please point out which commit implements the change? Searching for "Stencil" or "RenderStyle" yields no obvious commit like "Fixed Stencil renderstyle in OpenGL renderer". Neither is there anything in the changelog.

[Cherno]

IIRC, 3.6.0 should still have the old tint-style stencil model rendering.

[Graf Zahl]

I can't find it either. I think this happened as part of some internal refactoring in the hardware renderer.

[Korshun]

The change was introduced by commit 4604c9379c575d6b1a9eb94b9932af55e11a044a, which is a one line change. This line got another change in some commit after that. I reverted the following line (hw_sprites.cpp:1035 in GZDoom 3.7.2):

Code: Select all

if ((gltexture && gltexture->tex->GetTranslucency()) || (RenderStyle.Flags & STYLEF_RedIsAlpha) || (modelframe && thing->RenderStyle != DefaultRenderStyle()))

to

Code: Select all

if ((gltexture && gltexture->tex->GetTranslucency()) || (RenderStyle.Flags & STYLEF_RedIsAlpha))

and model recoloring is back and there seem to be no other bugs in the game.

[Graf Zahl]

You saw no other bugs because you didn't run into what prompted the change. It'd just bring back a bug with translucent models that weren't rendered completely.

[Nash]

The change was introduced by commit 4604c9379c575d6b1a9eb94b9932af55e11a044a

That commit was done deliberately to fix a long-standing issue where models would not render texture alpha correctly.

Before/After comparison:





In the "before" shot, the translucent parts of the texture simply get quantized to either a transparent/opaque state. In the "after" shot you can clearly see the smooth translucency from the texture's alpha being applied.