[gizdoom] Lazy palette shader

[Rachael]

Loving it. ^_^ Already making bug reports.

[JPL]

I've been using the version of this (or maybe it's separate code?) in GZDoom master lately, and it's a really great match of the software renderer look. I notice the skies, eg Knee Deep in the Dead's, get dithered a bit in a way they don't in ZDoom, is this easy to fix?

[Rachael]

Looking at the code - no it's not - dpJudas probably knows more about what's going on here than I do, I played with the shader side a bit though and it seems that the depth precision is off. Look out the window on E1M1 and notice how the sky is kinda whacky at the top - it looks normal with tonemaps turned off but it looks extremely palettized with the Palette tonemap on - and for the sky, other than the fading portion that only GZDoom does, anyway, it should look exactly the same.

So it seems to be a problem with the color-mapping generation code, but I don't know if that's in a shader or if it is in GZDoom's C++ code. If I had to take a *WILD* guess, it's just piggy-backing ZDoom's colormap code (the 64k one). I could be wrong, though, and I don't have time to look right now.

I played with the rounding on the shader, even removed the vestigial /4 calculations (creating a 0-63 ramp instead of a 0-255 ramp, though dpJudas may have inputted 0-255 for a reason), but I still got the weird look on the sky.

[dpJudas]

The sky is whacky at the top because gzdoom fades the sky to a solid color while zdoom does not. It runs the palette tonemap after other effects, which means stuff like the bloom effect (and ambient occlusion in the future) look roughly the same way as if the software renderer had attempted to do the same thing. If a fade to solid color was implemented for zdoom's skies, this is probably how it would look like.

About the shader itself, the way it works is that it builds a 512x512 lookup table as a texture (https://github.com/coelckers/gzdoom/blo ... s.cpp#L244). Then, the shader grabs a pixel from the screen and looks up which palette index color is closest to that color. It does this by first converting the color channel (red, green, blue) from a 0-1 floating point value to 0-255 integer and then shift off 2 bits so its rgb666 instead of rgb888. That value is then used as the index for the lookup.

I played with the rounding on the shader, even removed the vestigial /4 calculations (creating a 0-63 ramp instead of a 0-255 ramp, though dpJudas may have inputted 0-255 for a reason), but I still got the weird look on the sky.

My bit juggling skills are quite poor (same for reading hex values). So I did the shifting because it was easier in my head than realize that 6 bits is 64. Probably added some minor rounding precision loss by doing that. Oh well.

Generally, the output of the pass will never be a 100% match to ZDoom (the diminishing light effect moves in 32 discrete shades/steps). But I'm positively surprised just how close it gets.

[Rachael]

If I can figure out how to get this build environment working again, I will be posting a patch here soon - my suspicions were correct. You are piggybacking ZDoom's color matcher - which is not very accurate. It might be better to rewrite the colormatcher itself as a shader, but I don't have the skills for that so I am going to at least give you some C++ code on how I think it should be done. (It would be very similar to Lei's algorithm - imagine each color as an XYZ vertex and use Pythagorean formulas to calculate the closest points - it is not even necessary to calculate square roots, which will save you tremendously on processing time) - If you do rewrite my code as a shader, great, if not, no worries, I'll try to at least get it to compile.

I understand the sky won't look exactly the same - but most of it should look very close to how it did in Vanilla doom. It's the sky's own gradients (not GZDoom's) that are the problem.

What may actually be easier, is expanding ZDoom's own colormatcher to 18-bit (256k table) and then very little code change would be required. I am not too sure how happy Randi would be about that, though - but pretty much no system that runs ZDoom today will have any problem with it, especially since Win9x support was dropped in the last release - it would take an operating system THAT old in order to run any machine that would have an issue with a 256k table over a 32k one - the machine would be nigh unusable anyway with an OS ZDoom actually supports.

[torridgristle]

Will there ever be support for custom LUTs? It would be a lot more flexible than 2-colour gradients for screen effects, escpecially if multiple LUTs can be used and enabled from ACS.

[Rachael]

Alright. I got the algorithm correct - I think.

I am sorry, if I had more experience with C++ and OpenGL, I would port this directly to shaders myself, but I am better at writing algorithms than understanding the languages they are written in. If you have any questions about how this works or what's going on, please let me know, but I hope it's pretty obvious here. So the table generation is done purely in C++ - when I tested it on my computer it didn't seem too bad, someone else may have issues with load times though. (Hopefully not!)

---

Where: in gzdoom/src/gl/renderer/gl_postprocess.cpp, matches the line:

Code: Select all

PalEntry color = GPalette.BaseColors[ColorMatcher.Pick((r << 2) | (r >> 1), (g << 2) | (g >> 1), (b << 2) | (b >> 1))];

replace with:

Code: Select all

					// [SP] Custom LUT generation - this should be "better" than ZDoom's color matcher.
					int colDist, matchDist, colMatch, matchR, matchG, matchB;

					for (short i = 0; i < 256; i++) // i = index
					{
						matchR = r * 4 - GPalette.BaseColors[i].r;
						matchG = g * 4 - GPalette.BaseColors[i].g;
						matchB = b * 4 - GPalette.BaseColors[i].b;

						matchDist = matchR * matchR + matchG * matchG + matchB * matchB;
						if ( i == 0 || matchDist <= colDist ) // always match color 0, we'll find a better color anyway.
						{
							colDist = matchDist;
							colMatch = i;
						}
					}
					PalEntry color = GPalette.BaseColors[colMatch];

Test build containing said code: (built with FMOD only, and is 64-bit, sorry - drop in the official dev build dll's and it should work)

https://mega.nz/#!AUVF2TCB!z1hy8pUuMGiq ... ZXQ0Ve26MQ

With this build, check the sky in E1M1 - it looks a *LOT* better. Also as a side effect it improves the look of SHAWN2 textures as well.

(edit: I cleaned up the code a bit, opted to use long-ints instead of shorts to avoid issues with typecasting the distance checking formula)

Also - posted screenshots showing the difference - http://imgur.com/a/2lDSD

[ibm5155]

The performance impact isn't that big, running it at 1680x1050 only droped 5fps with my intel hd4000 (running at 45fps, with reflective floor, dynamic lights and a custom fog shader)
big screenshot

[Gez]

The color channels are on 6 bits so the max value is 63. If you compare them with 8-bit colors, the max value is 255. 63*4 = 252; 63<<2|63>>4 = 255.
47<<2|47>>4 = 190
31<<2|31>>4 = 127
15<<2|15>>4 = 31

Now you may say that with with values like 32 or 16 you'd expect to get 128 or 64, and you'll be getting 130 and 65 instead, but since the maximums aren't 64 and 256 it's not an accurate expectation. In floating point math, 32/63 = 129.524/255 (rounding a bit) so 130 is more accurate. You're not multiplying by just 4, you're multiplying by 4.047619047619047619047619047619...

Basically I think ZDoom's color matcher is fine but the line ought to be:

Code: Select all

PalEntry color = GPalette.BaseColors[ColorMatcher.Pick((r << 2) | (r >> 4), (g << 2) | (g >> 4), (b << 2) | (b >> 4))];

instead of right shifting by 1.

[dpJudas]

If I can figure out how to get this build environment working again, I will be posting a patch here soon - my suspicions were correct. You are piggybacking ZDoom's color matcher - which is not very accurate.

The code I used for generating the rgb666 table is virtually identical to the code zdoom uses to generate its rgb555 table (https://github.com/rheit/zdoom/blob/mas ... o.cpp#L657). That means if you could improve the quality of gzdoom's palette lookup, then you can do the same thing for zdoom's 18-bit table.

About the speed of the algorithm, that isn't very important in this case as it is a one time thing that only is needed if the palette changes. This only happens in a few rare situations: game start, restart ccmd, and some intermission screens with their own palette. Only if someone noticed a longer startup time would I make any attempts at optimizing it.

If this is causing a slowdown while playing (haven't checked), then that's a sign of a bug in the code where it checks if the palette is changing.

I would submit your changes as a PR already as you did it. Only thing is I'm not sure if it would be better to change ColorMatcher.Pick itself to do what your code is doing. My understanding of other parts of the codebase isn't good enough to make this determination (i.e. if Pick must do exactly what it does now for other reasons). Hopefully Randi, Graf or someone else more familiar with ColorMatcher will know.

Will there ever be support for custom LUTs? It would be a lot more flexible than 2-colour gradients for screen effects, escpecially if multiple LUTs can be used and enabled from ACS.

Tonemap palette will always just use the playpal lump. But it might possibly to roll your own if/when gzdoom gets support for custom screen space shaders in GLDEFS.

[Rachael]

Oh wow - Gez was correct. No wonder I didn't see any tables in the little bit of code I did look in. I wasn't really sure what was happening with the bit shifting and now I understand why it looked a little weird. I understood the concept of it, but wasn't sure why the particular numbers, themselves.

I compiled the original code with Gez's changes, and it actually works properly and it looks a little closer to the actual result I had made.

The code I used for generating the rgb666 table is virtually identical to the code zdoom uses to generate its rgb555 table (https://github.com/rheit/zdoom/blob/mas ... o.cpp#L657). That means if you could improve the quality of gzdoom's palette lookup, then you can do the same thing for zdoom's 18-bit table.

Believe me - I want to. But what's holding me back is if it is so easy why hasn't it been done before - I figured there's a philosophical reason in the way of it rather than a technical one (old system support). And that issue rests with Randi and Graf - most likely Randi.

I think it's fair to say after playing extensively with ZDoom's true-color matcher that a 15-bit(32k) table just doesn't meet our needs anymore, though. I'll see if I can raise the issue with Randi.

[dpJudas]

Believe me - I want to. But what's holding me back is if it is so easy why hasn't it been done before - I figured there's a philosophical reason in the way of it rather than a technical one (old system support). And that issue rests with Randi and Graf - most likely Randi.

I just meant that if there was a problem with ColorMatcher.Pick's precision, then that would have affected the rgb555 table used by zdoom as well. Seems Gez showed that is not the case and that it was just my lousy attempt at changing the original table generation code from rgb555 to rgb666 that failed miserably.

About changing zdoom to rgb666, the main problem here is that all the drawers need to be updated. They come in three flavors (C, 32 bit assembly, 64 bit assembly) that all have to changed and then tested. The change itself shouldn't be that hard, but it is not exactly a 5 minute thing either.

[Rachael]

Ah, I see.

And you're right - changing ZDoom to rgb666 does seem a bit harder than I thought it would be. I'm sure if I really stuck with it, I could do it, but I don't have that kind of motivation right now.

[Rachael]

So I have noticed there are differences in the results between my implementation of the algorithm and ZDoom's algorithm.

I've tried looking closely at the code to see what is different, at the very core they are the same, however ZDoom does not scan the full game palette. Which is okay for palette maps that have duplicated entries, as is usually the case - but in this case, it still seems to be gimping itself on what results it can get. I don't think it's actually iterating the full loop. With just the right lighting conditions, there are some glaring differences between the algorithm I provided and the one that was just fixed.

To test this, I built myself a copy of GZDoom with a custom CVar that allows me to toggle between the two algorithms. This is a pair of screenshots showing both: http://imgur.com/a/6NSfm

These are using gl_lightmode 4 and gl_fogmode 1.

I created a diff for this implementation, however it does not reset the tonemaps so a "restart" ccmd is required. (If this becomes permanent, I will make it reset the tonemap).

[dpJudas]

I barely notice any difference except that:

• Yours includes index 0 and 255 while FColorMatcher skips them. Thats full black and white, I think. FColorMatcher probably skips them because they may be functioning as a colorkey for transparent stuff in some situations.
• If the distance is the same between two candidates, then FColorMatcher picks the first candidate and yours the last. Since they got the same score, none of them is more correct than the other.

That's about it from what I can tell. This could affect the winner of a color if its near black or near white, because that's the two colors that FColorMatcher skipped. In all other situations I'd think they result would be identical (unless there's a lot of color matches that have the same distance).

You could try replace ColorMatcher.Pick with a direct call to BestColor (or add the 'first' and 'num' arguments to Pick and forward them). That will allow you to use the full 0-256 range with zdoom's algorithm to see if there's still any differences.