Can a GZDoom-like engine be made for Build games?

[Graf Zahl]

I could go into lengthy explanations here, but let's make this short. From what I have seen, EDuke32 has always been a project with a high turnover of extensive changes to the code but very little towards restructuring and cleanup. Although being compiled as C++, it's still pretty much 1990's code throughout - lots of large files - lots of large functions - lots (very lots!) of global variables - and lots and lots of #ifdefs - but no classes and no encapsulation.
Of course, any project being developed like this will eventually show some quality issues. Code cleanup is a vital part of keeping it manageable but from the looks of it, it never happened.

Any such code in ZDoom that got its life extended without addressing the fundamental problems it had eventually became unusable and had to be replaced in its entirety, the biggest offenders being the old menu code and the software renderer - where the latter one, despite the refactorings, still is very problematic because not everything could be properly cleaned up without breaking it. There's many other parts, too, which I cannot all list here, but there was a lot of code I have thrown out over the years and replaced it with something more modern.

[Kinsie]

I'm pretty sure eDuke32 was still using OpenGL 1.something right up until performance issues with the Ion Lastname preview release forced their hand.

[Graf Zahl]

If they have performance issues, it's mostly because their entire code base is rather shoddy (sorry to be blunt here, but this is a textbook example of what I consider clusterfuck source code)
If you have poorly structured code where it is hard to set up profiling mechanisms and there are difficulties doing partial refactorings becuase everything is interconnected it's not surprising that it develops performance issues - it has nothing to do with OpenGL1, 2 or 3. The Build engine works in a way that's even less optimizable than the Doom engine here. (Interestingly, the structural quality of their new Polymer renderer is not really much better than the old Polymost - it's also one single source file lumping everything together into one huge opaque wall of code where the graphics API access is spread all over the code - no chance to do targeted optimizations or test things that don't fit into the flow of how the whole thing works.)

Taking care of such issues takes time and even more willingness to toss out the garbage and rewrite it. Yes, this is partially the fault of Ken Silverman's coding style, but come on! After so many years there would have been countless opportunities to improve code quality. If you compare the source of the first GZDoom with current versions you'll see that the foundation is still the same, but lots of things have been reorganized or optimized over the years, e.g. fully abstracting the OpenGL backend from the actual hardware renderer.

[Graf Zahl]

Also, regarding EDuke32 in general, I have been sticking to a somewhat older version for some time and never bothered to update, but when I did the last few days I had the feeling to get a massive downgrade, not an upgrade. I have no idea what went wrong - can it be that with their entire focus on Ion Fury they somehow managed to compromise support for Duke3D? The old version from 2010(!) I was using before just felt a lot better.

Did you have any specific issues?

I noticed that over time the mouse aiming got worse and worse, but in the most recent builds it was very much improved. On the downside they shot the texture filtering option to shit. It suddenly disappeared from the menu and trying to manually re-enable it yielded some very shitty glitches. Now, many here may say they love their pixels and aren't bothered - but what's far, far worse than not having bilinear filtering is not having linear mipmapping and anisotropic filtering - without these options pixel soup is an inevitable result - and that just looks bad.

So I downloaded their entire repo as a Git conversion (can't do bisecting with SVN they still stick with) and started analyzing their changes.
The mouse was finally fixed on Feb 20th, 2019 and the texture filter system was broken on Mar 19th, 2019. I can't say I fully understand why this change was made, but Ion Fury sounds like a likely reason. So, I finally compiled myself a version from right before that change and that's clearly the best Build engine I have seen so far, it far surpasses the older versions with the weird mouse handling, it's also a lot better than BuildGDX for playing DN3D and it's certainly better than more recent builds. Let's hope they get this issue fixed again.

Now I got to port the mouse fix to NRedneck so that I do not have to use BuildGDX for that game anymore.

There's one other issue I need to find before I am satisfied: Just like any old SDL 1 application it forces the keyboard layout selection on the Windows taskbar on which is really annoying because it is normally bound to some system-internal hotkeys and tends to change on a whim sometimes. I wonder what kind of nonsense they implemented here to make this happen...

[Teddipetzi]

My issues were the same: changed mouse controls and the MIA texture options. It took a while until I realized that it was my compensation settings for their old and broken mouse code that made things go bad, and I changed that to something more normal now. This way the altered mouse is a vast improvement.

Thanks for the info about the dates when this happened, I just downloaded the last build from before the change you mentioned and that works perfectly, all texture settings are present.

If you can get the mouse issue in NRedneck fixed, will you post that EXE here? I could really use that, too.

[Nash]

Graf: any chance you'd be able to setup a fork of your changes? I'm interested in playing with your custom build.

[Graf Zahl]

I haven't made any changes yet. So far all I did was isolate the commit at which the regression happened so that I can think about how to go on and whether it is worth it.
I'll try to get as many more commits in as possible and then maybe do a fork.

While the renderer itself looks fine, the biggest problem I am having with the engine is performance. Even on a very simple map it is very choppy, especially right after map start. My guess is that all the crap was done to improve performance for Ion Fury, but since the render code is an organizational disaster of epic proportions (one single huge file at > 300kb and still growing) there's no good way to profile it and refactor into something more performant.

[Kinsie]

I noticed that over time the mouse aiming got worse and worse, but in the most recent builds it was very much improved. On the downside they shot the texture filtering option to shit. It suddenly disappeared from the menu and trying to manually re-enable it yielded some very shitty glitches. Now, many here may say they love their pixels and aren't bothered - but what's far, far worse than not having bilinear filtering is not having linear mipmapping and anisotropic filtering - without these options pixel soup is an inevitable result - and that just looks bad.

This is a known issue caused by their rejigging of the OpenGL renderer to make palette-swapped textures more accurate to the original software renderer. (Ion Fury still supports the software renderer and uses palette-swaps extensively for level texturing.) They plan on reimplementing filtering as a shader, but it'll apparently require more backend work.

[Graf Zahl]

TBH, doing filtering in the shader is yet another workaround of the problems caused by this convoluted code.

While the idea itself is fine, this should have been done more cleanly, but for that they'd first had to (gasp!) thoroughly refactor that 350kb mess of polymost.cpp into something clean and manageable. If they continue to heap new features onto it it will sooner or later break apart. But when looking at the commit history of EDuke I don't think this refactoring is going to happen.
Had GZDoom's renderer been in a comparable state we'd neither have a Vulkan backend nor multithreaded processing of the level's data. From the looks of it, how the Polymost renderer works, there's a huge potential for optimization - but what that needs first is a clean code base. The renderer performs quite complex calculations for every object it pushes to the hardware and IMO this could be easily serialized to 2 or 3 cores to increase the throughput - but like I said, it'd require some logical separation first.

Also regarding palette swaps, I haven't dived deeper into the code yet, but the naive approach for a palette change is to delete the texture with the first palette and then load in the second one. If the palette changes frequently, that's a first grade performance killer - something I noticed myself when dealing with the flashing and blinking enemies in Super Sonic Doom. My solution for the problem wasn't to revert to paletted textures but change the underlying system to be able to hold more than one hardware texture for each logical texture. That solved the performance problem without compromising everything. Of course for that you first need a cleanly separated texture manager, which brings us back to the code quality issues mentioned above.

All of this wouldn't be half as bad if they had a better working structure. (Yet another place where they stick to very outdated ideas.) There's no scheduled releases, the download always points to the latest automatic build, i.e. something like GZDoom's devbuild is all we get - all bugs, warts and temporary problems included, meaning they have no real chance for long-term planning because it'd destabilize the repo. I wonder if this is caused by SVN or that they still use SVN because of this forced linear workflow. - but in the end, since they have no branches and no planned releases, there is zero chance of any fundamental refactoring because it'd tear their entire system apart.

Imagine doing something like ZDoom's menu or savegame system rewrite in such an environment - it'd have rendered the engine unusable for several weeks!

So I'd expect that their problems will only pile up and not get solved. Anyone remember Doom Legacy...?
The only chance I see here to get a clean and functional code base would be to fork it, refactor what is needed and never look back, screw the missing features.


(Sorry, this has become a rant about software development philosophy, but I cannot deny that I have fundamental disagreements with how this project's development is organized.)

[Kinsie]

All of this wouldn't be half as bad if they had a better working structure. (Yet another place where they stick to very outdated ideas.) There's no scheduled releases, the download always points to the latest automatic build, i.e. something like GZDoom's devbuild is all we get - all bugs, warts and temporary problems included, meaning they have no real chance for long-term planning because it'd destabilize the repo. I wonder if this is caused by SVN or that they still use SVN because of this forced linear workflow. - but in the end, since they have no branches and no planned releases, there is zero chance of any fundamental refactoring because it'd tear their entire system apart.

Imagine doing something like ZDoom's menu or savegame system rewrite in such an environment - it'd have rendered the engine unusable for several weeks!

This has already been a problem. They elected to rework a lot of the collision detection code to avoid potential exploitation (ie. players clipping out of the map etc.) in the upcoming reworked multiplayer, and it rendered the game varying degrees of unplayable for at least a month. (This was especially funny since at least one of the devs got real pissy at Kaiser for rewriting the collision detection in Blood Fresh Supply shortly before this happened...)

[Graf Zahl]

I can't say I am surprised. It's precisely the stuff that's inevitable with how they organized their workflow.

EDIT: I think I am getting closer to the cause of the problem here: When transitioning to shaders they of course looked into the option to approximate the software renderer's lighting - but unlike GZDoom, where this was done with a full precision shader operating on true color input they apparently went back to the source, paletted textures and lookup tables and all - and at some point - unsurprisingly their aging OpenGL 1.x renderer could no longer coexist with those changes. I managed to graft 6 weeks of changes onto the version before they decided to cut off the ties so far - let's see how far I can get. As long as it doesn't involve fundamental rendering changes it should be ok - the further I can get the better.

[Phredreeke]

IMO it makes sense when you consider that the same system is also used for applying palswaps and that the game’s shade tables don’t have to be linear (which is what the old OpenGL renderer assumed)

[Graf Zahl]

IMO it makes sense when you consider that the same system is also used for applying palswaps

It depends on how you look at it and how you handle palswaps. GZDoom doesn't have the option to set a palette for a sector or such, but it allows palette translations on sprites and uses the same system for colored fonts. And like I said, one mod in the past went far enough with the translation system that the performance suffered, after that I implemented a better texture differentiation system and all was well again.

The downside of palettizing a texture in the shader is that it totally screws the texture filtering hardware in the GPU, requiring quite elaborate work in the shader itself. It's certainly not impossible, but it surely will cause performance problems for lower end hardware, which may have enough texture memory to hold 4 or 5 true color paletted variations of a texture set but not enough horsepower to run such a shader. That said, I'm still pissed at the GPU makers that they removed the paletted texture extension support 15 years ago - that extension was perfect for reducing the memory footprint of the used texture set but apparently didn't fit into the design patterns anymore. The good thing is that for a demanding game one can reasonably ask for at least 2GB of video RAM these days so the added textures aren't such a big problem.

and that the game’s shade tables don’t have to be linear (which is what the old OpenGL renderer assumed)

Correct. The original renderer had the same problem as GZDoom in its early years: Fixed function features just didn't cut it. In GZDoom I used exponential fog, not linear, and although I got quite close, it was nowhere near perfect.
The proper curve has to be calculated at some point, but I don't expect this to be any more complicated than the one being used to approximate Doom's original lighting.

The big problem here when using palette lookups is that it brings back all that ugly banding of the software renderer, and that's something I personally won't tolerate when playing - I already skipped on the new Blood release when I saw how it looked.

[crazyflyingdonut]

I have never delved too much into the technical stuff of the Build engine, but from what I have read online:

1. There isn't as much documentation on mapping for the Build engine as there is for id tech 1. And some of the documentation that does exist have a high chance of being outdated.

2. If you're wanting to make something, you'd be better off making it for id tech 1. Not only is software for it more up-to-date than software for the Build engine, it's also a lot easier to use.

[Phredreeke]

Blood uses a lot of different palswaps, something I learned the hard way about two years ago as I was making an upscaled texture pack for it.

As for banding NukeYKT (NBlood dev) came up with a way to interpolate between multiple shade table entries as to remove it.