QZDoom - ZDoom with True-Color (Version 1.3.0 released!)

[Rachael]

Uh, do you have to edit a ini file or something? Because AFAIK there's no option on the menu to use software render.

You're right - and I just checked GZDoom, and it's missing from there, too. It was probably removed.

You can use vid_renderer from either the console or the command-line - set it to 0 to enable the software renderer. The truecolor renderer right now uses 0, anyway (though I have tentative plans to possibly move it to "2" and use ZDoom's old paletted renderer as "0").

I'm probably going to create a menu that allows you to select the renderer, as well as the TC-renderer options (for filtering).

I can almost guarantee you Blade of Agony won't look well on it. But I'll tell you - it is my goal one day to be able to run Blade of Agony using QZDoom's software renderer.
EDIT:

Well - I guess Blade of Agony is not a TOTAL shipwreck... but I think QZDoom still has a long ways to go for it.

Lol I assume that's an old build of Blade of Agony because I removed all Doom inheritances or sprites in general for 3d models... But surely it looks good...

Btw keep up the good work Eruanna! (Still didn'thad the time to try this new port )

Yep that one's an oldie. I had it laying around when Torm was doing bug reports on GZDoom, and checked out my own copy at the time, since I had it handy I figured I'd test it. (Don't worry I downloaded a newer version later. )

Thank you for the praise!

[Edward-san]

MSVC problems

Did you try using clang with microsoft codegen for compilation?

[LkMax]

(though I have tentative plans to possibly move it to "2" and use ZDoom's old paletted renderer as "0").

Personally, I'd put 0 for standard software, 1 for colored software and 2 for hardware render, but that's just my habit of organizing everything in ascending order.

I'm probably going to create a menu that allows you to select the renderer, as well as the TC-renderer options (for filtering).

By that I assume both standard and colored? Having a option for that in the menu makes sense since you can configure quite a lot of stuff there already.

[Rachael]

Personally, I'd put 0 for standard software, 1 for colored software and 2 for hardware render, but that's just my habit of organizing everything in ascending order.

If vid_renderer had never been heard of before, I would too - however, since I included GZDoom in its entirety, I'm expanding on GZDoom's standards rather than creating my own and forcing it to fit - and GZDoom renderer switch keys off of vid_renderer being "1". Changing the way parent code works causes incompatibility issues with future code down the line and this project already suffers from quite a few instances of that - luckily, none of which have proven impossible to deal with yet. This decision will also make it compatible with GZDoom-compatible launchers down the line - they will continue to expect vid_renderer=1 means OpenGL and it will continue to function that way.

By that I assume both standard and colored? Having a option for that in the menu makes sense since you can configure quite a lot of stuff there already.

Yeah, I was going to have just a simple switch method of Paletted->OpenGL->TrueColor.

[Rachael]

So I figured out what was causing the crashes - and I tried to fix it, but I ended up making the problem possibly worse, instead, so I disabled the feature that was causing it by default temporarily, for now.

So for now - multi-threading is not active. It can be turned on forcefully by setting "r_multithreaded" to true, but beware that it is not stable right now. I added an "r_multithreadedmax" to try and keep a sane limit on the number of threads in queue but I've found out that it really doesn't do anything useful at the moment. (Still a noob at C++... lol... oh well)

Anyway, when this problem is fixed, I'll have multi-threading on by default, because it is useful and it does speed things up, but it's pretty pointless if it causes crashes right where it's really needed.

[dpJudas]

Did you try using clang with microsoft codegen for compilation?

No, I didn't check if clang is better at it. But even if it was, part of the problem is that C++ is not a very good language to do the drawers in. This is because the C++ const keyword is more or less irrelevant to the compiler, the constexpr C++11 keyword is sort of a joke that the compilers may or may not use for anything. Pointer aliasing and forced function inlining may not even happen if you use compiler specific keywords for it. And almost all functions are public in C++ due to a terrible linking model. Last, but not least, vector types are not first class and they are critical for good performance in 32-bit drawers.

The classical solution is to switch to assembly, possibly with some macro language that helps output all the different permutations. The problem with that is just 1) I really really suck at hand optimizing, 2) even with assembly there's a large difference in the vector feature set between processors (MMX, SSE, AVX, etc.). So I'm choosing a different path that was unlocked a few years ago: LLVM JIT compilation.

I already have the span drawers working in all its blend modes, using the following code to emit the LLVM bitcode: https://github.com/raa-eruanna/qzdoom/blob/llvmcompiler/src/r_compiler/fixedfunction/drawspancodegen.cpp. It uses a trick where constructors and operators in C++ classes generates the LLVM instructions, allowing the code I linked to almost look like a normal program. The key difference is that all the variables are immutable (SSA, single static assignment) at the compiler backend level, which should give clang's backend compiler its best possible kind of code to heavily optimize. This is then compiled on the fly when QZDoom boots, allowing it to know exactly which CPU it is targeting.

I'm probably a day or two away from having the wall drawers ported as well. At that point I'll be able to compare frame rates directly to the old code. Already at this point I'm very happy with how its turning out.

[dpJudas]

So I figured out what was causing the crashes - and I tried to fix it, but I ended up making the problem possibly worse, instead, so I disabled the feature that was causing it by default temporarily, for now.

Which feature is crashing it? If you have some point in code where it is critical the workers aren't doing anything, you can call DrawerCommandQueue::WaitForWorkers() to ensure any queued work has ended.

[ibm5155]

Idk how this is done, but you guys could limit The number of threads by the number of cpu cores or the number of cpu cores * 2...
Out of curiosity, are you guys using that c++ sdt::thread? or a specific thread struct for each os type?

[dpJudas]

It uses std::thread::hardware_concurrency() to pick its number of cores. std::thread for its threads, std::mutex and std::condition_variable for thread synchronization. Luckily C++11 removed the need to use platform specific stuff for this unless you have very specific needs. See DrawerCommandQueue::StartThreads for the details.

[Gez]

Personally, I'd put 0 for standard software, 1 for colored software and 2 for hardware render, but that's just my habit of organizing everything in ascending order.

If vid_renderer had never been heard of before, I would too - however, since I included GZDoom in its entirety, I'm expanding on GZDoom's standards rather than creating my own and forcing it to fit - and GZDoom renderer switch keys off of vid_renderer being "1". Changing the way parent code works causes incompatibility issues with future code down the line and this project already suffers from quite a few instances of that - luckily, none of which have proven impossible to deal with yet. This decision will also make it compatible with GZDoom-compatible launchers down the line - they will continue to expect vid_renderer=1 means OpenGL and it will continue to function that way.

-1: Old software
0: Software 32 bit
1: OpenGL

[Rachael]

-1: Old software
0: Software 32 bit
1: OpenGL

That - actually - might work.

So I figured out what was causing the crashes - and I tried to fix it, but I ended up making the problem possibly worse, instead, so I disabled the feature that was causing it by default temporarily, for now.

Which feature is crashing it? If you have some point in code where it is critical the workers aren't doing anything, you can call DrawerCommandQueue::WaitForWorkers() to ensure any queued work has ended.

Queueing up too many threads crashes it. Revert these two commits, compile, and load Ozymandias81's Gore. That mod is an extremely reliable way to check this - just load the mod and watch the title screen.

[dpJudas]

The command queue isn't the number of threads. It is a list of commands/tasks (DrawerCommand::Execute) it should run/execute on each thread.

There's already a limit in place for how many can be queued. When AllocMemory returns null it empties the queue by running all the threads (queue->Finish()) on the queue and waits. I've intentionally made it so that should not happen for any reasonably complex map (memorypool is 16 MB). Ideally there would be no limit at all, but for performance reasons the system uses its own memory pool rather than 'new' as there are thousands of commands (one for each column or span to be drawn).

In detail, the algorithm is as follows:

1. The "upper" part of the renderer (R_RenderBSP and friends) draws columns and rows by calling colfunc, vline4, and so on.
2. Instead of drawing a column in colfunc, it queues a command object with the information to do so, using DrawerCommandQueue::QueueCommand.
3. When main renderer thinks it is done drawing the scene it calls DrawerCommandQueue::end():
3a. The end() function awakens all the threads (7 on my computer) that then all run through the list of queued commands.
3b. Each thread does a "for (auto command : queue) command->Execute(core, numCores);" where core is the index of that core, and numCores is how many threads we have.
3c. The actual work is split between the threads in the drawers themselves by assigning every N line to each core.
3d. When all threads finish executing the queue, the end() function returns.

If it crashes with multi-threading on, it must either mean there's a missing DrawerCommandQueue::end() call somewhere, or there's a buffer overrun related to core or numCores. The code relies on the fact that all pointers passed into colfunc and friends must remain valid until the queue is flushed. For example, when it renders a canvas texture it must finish the threaded drawing before passing on that texture object to rendering by the main scene.

[Rachael]

I'm going to guess there's a buffer overrun, then, because I was still able to get problems in Gore, even with multithreading off - it just took longer to get them.

[Rachael]

New version posted. This focuses on Hexen fixes. The skies in Hexen were not rendering properly (multi-layer skies would not show their behind layer), and it also adds fog to levels that had a fogmap defined. (The "fogmap" is static at the moment - sorry - I'll improve on it later - regular fog should still work just fine, though)

In particular, both changes should be noticeable in the map "Sacred Grove" (Map11), and "Orchard of Lamentations" (Map32)

[Rachael]

New build posted. This one pulls all changes from both GZDoom and ZDoom from so far today (or "yesterday" if you're in a European/Australian time zone).