Very bad performance with FX-heavy scenes

[Rachael]

One thing you can do, but this will take a bit of coding, is to have all weather particles become +NOINTERACTION and simply have them fade out after a certain time. It will take some tweaking in the map to figure out when, but that will reduce a lot of actor-to-actor physics processing. Keep in mind that you are not off the hook with this flag - the actors still must be destroyed, or they will cause even greater issues.

EDIT: @Tormentor667: Here's a debug build for you -
https://mega.nz/#!kFsSjLKR!h4tjNxYSlG4U ... 9NVYNWaYpY

Oh yes, that really sucks now - unfortunately it sucks that hard that even C1M1 isn't playable anymore

Now you know what we deal with when we go bug hunting.

Anyway the point of me giving you that was so that you can actually measure your frame rate. If it dips below a certain amount, then you know that you're in a problem zone. But where that threshold is - that's up to you and your team to decide.

[Ozymandias81]

While I am actually a donkey when it comes ZScript, could ThinkerIterator imply more "think" operations for the engine, so augmenting the amount of thinkers unnecessarily? Some actors on BoA, such as AlertLights, uses it...
Also could be even the fact that on BoA we have some decorate to zscript inheritances another cause of high amount of thinkers in general, and if yes how this could be solved code-wise?

[Rachael]

Yes - ThinkerIterator can be bad to use on maps with lots of actors - especially if the excessive number of actors use it themselves - this would cause each actor to individually loop over every other actor themselves - leading to an exponential amount of processing. In C2M5, for example - let's pretend every actor used it - that's 225 MILLION (potential) iterations PER TIC just for the actors alone! And while good beefy CPU's can handle that - that's right about the point where they start to buckle. That's nevermind the other stuff - like your programs running in the background, GZDoom's scene setup, and other things that must happen before the frame gets fully rendered.

You basically have few options at this point. The best option right now is to simplify your weather processing so that it is a single texture overlaid over the player's screen while the player can view them. Creating tons actors for your weather is a bad idea on multiple fronts - especially with the number of actors spawned here. A lot of what you want to do can be done simply with intense fog and at most 3 or 4 large sprites projected out in front of the player. Doing a whole entire actor for each individual snowflake or rain drop is what's causing your performance issues here - and that kind of thing will not go away with time because CPU's are not going to increase in clock speed in the coming years. Currently, GZDoom can only do single-core processing for its actors, or else it will break netgame and demo compatibility.

The only hope you have of continuing to use the system as it is now is if GZDoom goes fully C/S and, later, starts multi-threading its actor processing - then, any CPU-specific issues can be fixed over time with simply more CPU cores. But that's something that's still on the horizon, and the possibility of that happening should not be immediately developed for - and especially not for a mod with this kind of graphic fidelity.

For steam - you may consider using a few wallsprites and summoning a few puffs here and there, rather than a single actor for every single puff.

[Tormentor667]

Thanks alot for that detailed information Rachael. I have one question: Would it make sense to simply split up C2M5 into two maps, where the first part is the introduction and train sequence until you enter the facility and the second part is the facility itself? Would that double the framerate at least?

[AFADoomer]

While I am actually a donkey when it comes ZScript, could ThinkerIterator imply more "think" operations for the engine, so augmenting the amount of thinkers unnecessarily? Some actors on BoA, such as AlertLights, uses it...

This is why most of the functions that I've added that use a ThinkerIterator are either called only in PostBeginPlay (so, once on actor spawn), or are only called every certain number of tics.

That said, I may be able to further limit the impact... Right now, I'm not passing a statnum parameter to any of the thinker iterators, so each of them loops over *all* thinkers instead of those with a specific statnum. I might also be able to assign certain actors a different statnum, and use that to further limit how many thinkers are iterated over.

[Graf Zahl]

About C2M5, there isn't one specific thing that seems to cost this much time. There's over 7000 things on the map to begin with and several of those are constantly spawning other actors. It just adds up to a huge amount.
I'll have to do some more detailed profiling to see what items cost the most.

[Rachael]

That is true - it's a number of things on that map. But I do believe even one of them being reduced would help things a lot - and the weather and steam are the most noticeable ones.

7000 is just a bad number of things to have starting on a map, period. I'd consider splitting the map in two, at that number, anyhow - the trenches leading up to the train depot, and then the train depot itself. There's no shame in marking them both as hub maps if you need to go back and forth between them (I have not actually played that map all the way through so I don't know - I just noclipped around and explored it).

Rex had to do this once in Paranoid. Initially MAP04+MAP08 were all one map, but for CPU's that were available at that time it had the potential to grind performance to a halt. When he realized he could split it in two - he did. This was also combined with A_CheckSightOrRange pulled into ZDoom and used on the SFX spawners, and improved performance dramatically.

[Graf Zahl]

Doing some profiling on TickThinkers:

By far the biggest time waster is the player. And from the looks of it it is indeed several ThinkerIterators, which, with such a large number of thinkers on the map just add up. The player alone is roughly 1/6th of the entire think time.
The rest is just the sheer amount of actors on the map. It has to be reduced. Each one alone is ok, but it just adds up.

[Graf Zahl]

Something interesting: ZDoom's actor carry code is responsible for approx. 1/3 of the entire think time! I verified this with multiple maps that contain scrollers and the result is very consistent. Once there is even a single actor carrying sector in the map, performance goes down quite dramatically. I think the code in question needs some serious optimization.

[Graf Zahl]

Fixed the scroller stuff to be less of a performance hog.
Although it should be noted that the change may be noticable if some scripted actors move other stuff around that may depend on scrolling sectors. Highly unlikely but you never know. It's still the lesser evil in this case, the performance gain is far, far more important here.

[Rachael]

Is there any chance I could ask you to commit your profiling code to a branch for future usage in other projects?

Will test the scrollers - I have a few favorite maps that use them, so I can at least see if they create any issues as of the last commit.

[Ozymandias81]

Thank you kindly Rachael and Graf: so the best plan to optimize BoA now would be to re-elaborate some lines for BJ code, nitpick all actors and go for a split map? BTW I'll come with some screenshots just to show how c2m5 act for me on my very low-end machine.

[Rachael]

Any one of those will improve things greatly, two would be even better, three may not be necessary. I guess just pick and choose which two prospects your team likes the best and run with it and see how it goes.

I can compile another Debug build for Tormentor if performance testing is needed, on his end - although that's not really what Debug builds are good for, they at least help figure out some of these problems.

[Graf Zahl]

Is there any chance I could ask you to commit your profiling code to a branch for future usage in other projects?

No. It was just a very crude and quick hack that created a large static array of cycle_t's, indexed by the class type name. I already deleted it. It didn't help much anyway.
The only thing I noticed that a large portion of time was evenly distributed across all actors, so I selectively started commenting out parts of AActor::Tick until I found something that could be optimized.

There's still too much time being spent on non-moving actors so it may make sense to add a few specific checks to optimize these common cases a bit

[Nash]

Curious... what exactly is so bad about the BoA player pawn? Would be useful for educational purposes (AKA ZScripting best practices or something like that).