Dynamic Array Efficiency?

[kb1]

release your source port kb1

Hey, Kinsie - I could very well be a happy man if I knew your motivations behind that command...
It's kinda funny: I've been asked to release my port by a lot of people, without any of them having a clue about exactly what I'd be releasing. Not sure what everyone wants or expects...but, I suppose it had better be a quite good lump of code, huh?

[Major Cooke]

Heh, alright, you got me. Took me long enough to figure it out. Here' I'm thinking you're being serious...

I was in fact being completely serious. I wasn't joking around. Why you continued to explain yourself when I made it pretty clear what I was going to do is beyond me. You didn't need to keep explaining yourself.

Furthermore, I talked to several people on Discord who know and have delved into the source code. They said this stuff is more likely to slow things down than to speed it up, and that if an array consists mostly of nulls, it should probably have its size reduced. Some of them actually advise using my old stuff where I push to a temporary array and then move it over, as they think this iteration stuff will most likely be slowing down more than speeding it up.

So many things to consider. Sheesh... So torn on which one to stick with, but honestly, since they only happen once every X seconds, I think this is one of those things where it can safely go either way and have no performance impact since they both work relatively well due to happening so infrequently.

I have yet to decide fully which one I'm going for. Trying to measure which one is best is almost impossible, unless I were to try and constantly run them every single tic I suppose.

[Apeirogon]

Im lost point of what you try to achieve after second page of discussion, soo....what you try to achieve? Are actions with arrays really slow? You definitely dont want increase array efficiency just to increase array efficiency.
Also said graf in the first post "if actions with array causes performance issues, you use it wrong". So probably you use them wrong?

[Major Cooke]

Don't worry about it Apeirogon.

[kb1]

Im lost point of what you try to achieve after second page of discussion, soo....what you try to achieve? Are actions with arrays really slow? You definitely dont want increase array efficiency just to increase array efficiency.
Also said graf in the first post "if actions with array causes performance issues, you use it wrong". So probably you use them wrong?

The way I understand it, Major Cooke is providing a library of pre-built ZScript tools. One of those tools provides quick access to a user-defined subset of a map's actors. This is important, because, done incorrectly (iterating through thinkers, for example), performance issues can occur.

Heh, alright, you got me. Took me long enough to figure it out. Here' I'm thinking you're being serious...

I was in fact being completely serious. I wasn't joking around.

Oops, sorry about that.

Listen, I can't speak for others. But I know what I wrote probably delivers, and that it probably wails. (profile, as always). But, it's a system - don't expect it to work piecemeal. I continued to explain, because I want you to have the best - that's the gig, for me. Any less is... less. It's just not fun for me otherwise.

So many things to consider. Sheesh... So torn on which one to stick with, but honestly, since they only happen once every X seconds, I think this is one of those things where it can safely go either way and have no performance impact since they both work relatively well due to happening so infrequently.

Are you sure about X second intervals, and not sooner? I ask, because: Isn't this for a library that others are going to use, at random intervals to parse through the active actor list? You can't depend on them waiting X seconds, can you? That's why I've been concerned with performance. I've been assuming that a library user might wish to run through the actors, say, every other tic, which is why it's so important to keep *their iterations* minimal and blazingly fast...not just *your* Clean function's iterations. Maybe you see the library being used differently?

I have yet to decide fully which one I'm going for. Trying to measure which one is best is almost impossible, unless I were to try and constantly run them every single tic I suppose.

If you can forego some accuracy, and you just want to compare the performance of two similar scripts, this might do the trick:
1. Make a simple map with 2 very large rooms.
2. Edit the map by adding 2500 imps.
3. Save map, and start up the game.
4. Play the map, and turn on FPS counter.
5. Repeat steps 2 thru 5 until the FPS dips below 35.
6. Take out 1000 monsters until the average FPS goes back up to 35.

You should now have a map that just barely starts to lag on your computer. Now, you can add your ZScript library. Any custom function that uses actors in this map is going to be *really sensitive* to the performance of your code. In other words, even a slight slowdown is going to be greatly multiplied by imp count + imp missile count. Imps are good, because, as they fire missiles, the actor count is going to go way up. And, when the missiles explode, you're going to get *a lot* of NULLs scattered within your arrays.

*And*, if you use a command that does a "kill all", and you happen to be doing something like SlotCount, those iterations should quickly drop all the way to 0, even while the array still has thousands of elements. That's what I've been describing for multiple pages.

You know, most likely, your library will be really fast, anyway. But, someone will always find a way to write really slow code using your fast library! There's nothing wrong a a little bit of performance tuning.

Anyway, if you decide, one way or another, and you happen to feel like doing some performance profiling, drop a reply here if you would. Thanks.

[Kinsie]

ship the source port, friend

[kb1]

ship the source port, friend

Heh, would you like fries with that?

Seriously, *everything* I am doing is leading up to me being able to get back to my passions, including my port. I'm on it.

[Major Cooke]

Are you sure about X second intervals, and not sooner? I ask, because: Isn't this for a library that others are going to use, at random intervals to parse through the active actor list? You can't depend on them waiting X seconds, can you? That's why I've been concerned with performance. I've been assuming that a library user might wish to run through the actors, say, every other tic, which is why it's so important to keep *their iterations* minimal and blazingly fast...not just *your* Clean function's iterations. Maybe you see the library being used differently?

Yes. The interval is only for triggering the cleaning. Event handlers have functions that are called which I utilize to record actors.

WorldThingSpawned does the initial assignments for whenever something spawns.
WorldThingDied will find the actor in the array if it's a monster and move it to the DeadMonster array, leaving a null in the old spot.
WorldThingDestroyed, thankfully isn't necessary because the reference is automatically set to null inside of the arrays.
WorldThingRevived will reset the monster back to the live array.

Sadly, WorldThingRevived does not take into account respawning, so it's also important to handle every interval (X seconds, currently 5).

[kb1]

I don't think you caught my drift: I'm not talking about the internal workings of the class, I'm thinking about all the neat things a coder might want to use your class for:
For example, I might want to spawn sparks off of all fireballs. Or, I might want to add a health bar to all bosses, or all monsters.

You've claimed that you're offering this class as a general-purpose tool (or, maybe I just read that into your purpose - if so, I apologize). My point is, you cannot know how people will use the array (which is, basically the purpose of the class). Your users may want to iterate through the class each tic. Or maybe twice each tic. Therefore, performance is of the essence. Therefore, any technique that reduces iterations should be of utmost importance (if you want your class to be successful). And, you don't get performace by forcing the array user to parse through a chunk of NULLs each time. I mean, why even pack the actors if they need to parse through the NULLs anyway??? It doesn't make any sense. Without the SlotCount, the code that so carefully packs the actors becomes totally unnecessary, because you're going to walk through all the NULLs anyway.

Come on, man - think it through. I guess I shouldn't feel so invested. I think you're smarter than this - that's why I can't stand you throwing away all the benefits.

Again, it's yours to do with what you want. Personally, I cringe when I hear people say "it's fast enough", because there's always someone with a slower PC. And lag time is no fun for anyone.

[Major Cooke]

It's actually kinda funny you say that. Some folks in Discord have actually stated your code is more likely to cause slowdowns with all these iterations, and they've worked on/with the JIT compiler code that helps power ZScript.

Which is why I'm still torn on this whole thing, because their word carries weight.

[Graf Zahl]

I'd take the first rule of optimization here: Any optimization that complicates the code is not worth taking unless it has a definitive and measurable impact on overall performance.

[Major Cooke]

Which is the primary reason I don't want to use SlotCount. It's just more maintenance that seems needless.

In fact, because of the iteration stuff as a whole, some have even suggested that my old code is probably more optimized with that for loop, which pushes any non-nulls to a temporary array and overrides the main array.

[kb1]

I'd take the first rule of optimization here: Any optimization that complicates the code is not worth taking unless it has a definitive and measurable impact on overall performance.

Exactly. Which is why the only word that should carry any weight is a profile of all versions in question.

Which is the primary reason I don't want to use SlotCount. It's just more maintenance that seems needless.

In fact, because of the iteration stuff as a whole, some have even suggested that my old code is probably more optimized with that for loop, which pushes any non-nulls to a temporary array and overrides the main array.

Maintenance? You mean Copy and Paste? Test them. Or don't.

I mean, I did the hard part, by writing the thing. All I've ever asked is that you profile it in original form, possibly alongside some other versions.

The timing is the only word that holds worth. And, yeah, sure, I could test them, and I may get to that point. But, I gotta tell you, this is not my project - it was just a good faith gesture, and it sure wasn't worth all of this.

[Graf Zahl]

I'd take the first rule of optimization here: Any optimization that complicates the code is not worth taking unless it has a definitive and measurable impact on overall performance.

Exactly. Which is why the only word that should carry any weight is a profile of all versions in question.

No, that's not the only thing that should carry weight.
First, code should only be subjected to hard optimization if there is an actual performance concern.
And in cases where code readability and maintainability is involved the most important thing is to keep the code in a state where it can be refactored.

I have lost too much code over the years to unnecessary optimization.

[kb1]

I understand what your saying. But there's nothing unusual or difficult to understand in the code I submitted, as far as I know. I'm dumbfounded by the amount of discussion and scrutiny that tiny class has been through. The whole point of having a library is to contain a set of useful functions that have been tested and known to work, so the wheel doesn't have to be reinvented each time a similar task is asked for. Once it is trusted, it can be dropped into any number of projects, and used as-is.

This, of course, means that the scope of the code is expanded beyond one map, or even one mapset, to, potentially multiple projects.

To that end, the library should be
* documented
* easy to use
* fairly efficient
* trustworthy

Graf, I'm sure, with your experience, you recognize the technique. There's nothing complex about it - it's a single-pass item compact, using standard features of your language. It's written to perform well and scale well, but hard optimization? That's a bit of a stretch, isn't it? There's a difference between clean code and unnecessarily reading unused space over and over - that doesn't make the code easier to read, that makes me think "Why in the world is he reading unused elements over and over...is that necessary?"

Anyway, the stated goal was "minimum iterations", and that's what I provided. Of course, you could absorb my code and produce an intrinsic function Array.Pack() which would solve the issue and be faster, yet.

@Major Cooke: If having the SlotCount variable really bothers you that much, replace it with a resize of the array, and use Size() in place of SlotCount - it should perform very similar. You can even Push() if you use that technique.

I'm sorry to say that me trying to do something nice here has had a net negative impact on my life, so I think I'm done. Take care.