ZDoom

[quote="dpJudas"]Perhaps the best solution is to do both - if there is a brightmap and the color is above some threshold (let's say 0.1) then it won't be subject to SSAO. And if there's a nossao specified in the gldefs file the entire texture is excluded.[/quote]
That sounds a very user friendly way to do it (noting Graf's comment, of course).

Can we please wait with any changes here until I am done refactoring the renderer for the Vulkan backend? Any change here right now would create some inconvenience because I shuffled around a lot of code in the last few days.

Using the brightmap can also be done, but that would mean it doesn't work on traditional Doom light textures where they can be on and off using sector light.

Perhaps the best solution is to do both - if there is a brightmap and the color is above some threshold (let's say 0.1) then it won't be subject to SSAO. And if there's a nossao specified in the gldefs file the entire texture is excluded.

I do like the idea of being able to derive from the brightmap. That way examples like the model lamp could have a nice bright dome but SSAO shadows could be applied to the non-bright areas of the model.

Yeah, for PBR, the AO map could be used to define where SSAO is "allowed" to appear (so a fully white pixel means absolutely no AO drawn), but for diffuse/spec/norm, I was wondering if it's a good idea to derive from the brightmap.

Doesn't PBR include an AO map, defining where AO can be applied and the strength of it?

Which is a better solution - nossao property, or have the SSAO pass automatically reduce SSAO intensity with brightmaps? The latter makes more sense physically, and would give more control to the artist.

The short answer is that currently there is no way. While what Rachael suggests will reduce the effect slightly, it will also tell the bloom pass that woah this pixel is really bright (it will most certainly bloom).

The SSAO pass itself does actually support excluding pixels from SSAO. It is used by portal entrances to reduce the visual transition between each side. I could add a nossao property to the textures in gldefs that would do what you're requesting.

vec4 Process(vec4 color)
{
	vec2 texCoord = gl_TexCoord[0].st;
	return vec4(getTexel(texCoord).rgb * 1.33333, getTexel(texCoord).a);
}

hardwareshader texture BRIGHTEX { Shader "FakeHDR.glsl" }

The most effective way I really know of is to do a pseudo- high-dynamic range on the textures.

For instance, you can put shaders on the textures (if your mod is intended to be GZDoom-only) like this:

FakeHDR.glsl:
[code]vec4 Process(vec4 color)
{
vec2 texCoord = gl_TexCoord[0].st;
return vec4(getTexel(texCoord).rgb * 1.33333, getTexel(texCoord).a);
}[/code]

gldefs.txt:
[code]hardwareshader texture BRIGHTEX { Shader "FakeHDR.glsl" }[/code]

Then - your effective colour range is from 0-192 (where 192 is full white, but with SSAO) - and everything brighter is actually brighter. The colour actual values when rendered is, in fact, stored in floats, so overbrighting it will actually work.

Can anyone suggest a way of combatting this?

I have noticed that, sometimes, SSAO will draw shadows in areas where I would rather it wouldn't. For example, here's a picture that I posted before:

[img]https://i.imgur.com/hoN9jid.jpg[/img]

The texture in the alcoves is pure white and the light level is 255 yet there are still shadows in the corners. While this particular example is just that, an example, it's common to use textures such as LITE5 in bright alcoves for computers or whatever and this effect makes the LITE textures look far duller than they should. I've tried brightmaps and dynamic lights to try and counter the effect but not really had any success.

Speaking of lights, here's another situation with the same basic problem.

This model lamp, when it's out in the open is nice and bright. It has a dynamic light attached and a brightmap makes the dome nice and bright too. The dome looks like it's meant to - like a bright light source.

[img]https://i.imgur.com/WuranxW.jpg[/img]

However, place it near a wall and the bright domed light source gets a shadow on it. That's definintely not required and makes the light look odd.

[img]https://i.imgur.com/23YveMq.jpg[/img]

Of course, in this particular case, I'd like the dome to be bright but other parts could legitimately be affected by shadows.

So, is there any way to combat this? Or is it just, as Graf would put it...

[quote="Graf Zahl"]Making SSAO not glitch is virtually impossible. I already disabled that feature entirely for playing because I found far too many maps with high detail geometry where the drawbacks far outweighed the benefits[/quote]

[BTW, I put this in assets rather than mapping because it is to do with both models and mapping and I suspect that, if there is any solution, it my come from making particular assets (textures or parts of a model skin) immune to SSAO - however, I suspect there simply won't be a solution.]