ZScript Discussion

[dpJudas]

I don't think he meant quite that.
Imagine we have an abstract def add_vertex(v).
Now, we already know that add_vertex is something that's supposed to add a vertex and that v is probably a vertex.
But what type v is and how do I call this function? In a language that has static typing, the function would look like bool add_vertex(Vertex v), which would hint me that the function can return true/false and takes a parameter of type Vertex, which (in a proper IDE) could be looked up directly by following the declaration. Feel the difference.

It actually goes much deeper than that. In a language like C++ I have virtually a mathmatical guarantee that Vertex has three floats (not doubles, not integers, and especially NOT strings). There is no extra secret v.hackMessageToTheTriangleDrawer = true that is checked 4 levels deep in the implementation, because someone didn't want to refactor something. If the function says 'bool' as the return type, I already know this function is fishy and I need to examine the implementation to see what hack made it return a bool.

Now, I'm sure Eevee is just eager to state that theoretically anything can be cast to anything in C++, but the difference is that the likelihood of another developer doing this is very very low. The hackMessageToTheTriangleDrawer is almost considered a feature by dynamic code fans, which means the less skilled developers will do this. And when they accidentally pass a string into the add_vertex class, they will "fix" add_vertex to support strings. Woohoo. Now all code needs to be aware there can be strings in the data.

You can also take the const keyword in C++ as an example. This keyword means nothing to the compiler backend - any code in C++ can at any point cast that. The const keyword on a function also means little as there might be a mutable variable on the class, or a const_cast of 'this'. But it sends a very strong message to the developer reading the code that it is supposed to be an immutable operation. It states intent that can only otherwise be stated via actual documentation. Actual documentation does not exist (in any language) for the codebases I have been working on.

[Gez]

Whoops, I edited my post with some Swift syntax while you were replying. But yes, null is a "separate type" in Swift and Rust and others — or, rather, "nullable T" is a separate type, and a plain T can never be null.

It's a good idea for a general purpose language; but I'm not sure it'd be useful in ZScript, given that the main offenders here (an actor's target, tracer, and master pointers) would definitely have to be nullable since under the hood they are C/C++ pointers which are null more often than not. And that's something that's been true since the id/Raven codebase days, it's not something that ZDoom invented much later.

[Graf Zahl]

Now, I'm sure Eevee is just eager to state that theoretically anything can be cast to anything in C++, but the difference is that the likelihood of another developer doing this is very very low. The hackMessageToTheTriangleDrawer is almost considered a feature by dynamic code fans, which means the less skilled developers will do this. And when they accidentally pass a string into the add_vertex class, they will "fix" add_vertex to support strings. Woohoo. Now all code needs to be aware there can be strings in the data.

Heh, yes. I've seen stuff like that, too. Like I said: The lack of typing opens a completely new category of potential errors, which not only increases testing requirements but ultimately complicate matters rather than simplifying them. For god's sake I have a computer at my disposal, so please hand be the tools to use that power to prevent accidental errors from going unnoticed!

In an untyped language the computer often has no knowledge if some construct may be legal or not so it has to accept it as potentially valid. A good typed language errors out as soon as one tries to compile the stuff, long, long before it hits any critical point.

And one word about non-nullable pointers: C++ is good enough to implement something as well, but ultimately it's not about non-nullable pointers but about the question if those fit the bill. And here they clearly do not. These pointers CAN be null, and if the code forgets to check this, it's better to get a runtime error than silent failure. Silent failure is always the worst outcome for a programming error, because - good luck finding that! It's the stuff nightmares are made of - I'd rather see a good hard crash I can track down than some code misbehaving and not being able to find out why. I really find it disturbing that some people consider that an asset.

For me:

- compile time errors are better than runtime errors.
- runtime errors are better than silent failure.
- the worst errors are those when code several miles away from the actual cause breaks. Anything that makes it harder to write such code is a good thing. And the more lax a language is the more likely this can happen.

[Major Cooke]

Depends on the relation to what you call "platform independent".
Another option is MonoGame, which comes with OpenGL support out of the box and requires installing a 33 Mb redistributable to run. This will theoretically allow running ZDoom on iOS, Android, MacOS, Linux, all Windows platforms, OUYA, PS4, PSVita, and Xbox One.

This sounds quite intriguing.

[Major Cooke]

So, this commit... What will it mean? As in, what will change in how we use virtual functions?

[ZZYZX]

Unity uses C# (and .NET 2.0) for scripting and it's quite crossplatform. Just saying. Sorry for the late input
It mostly uses Mono, for some platforms it also uses complex chain of il2cpp->llvm->(target platform asm here), for example, when compiling for WebGL/JS, as there is no direct way to run C# on JS.

Although it does have minor problems on some platforms (e.g. 64-bit Mono, at least the one packaged with Unity, would randomly throw IndexOutOfRangeException while accessing valid array member).
I call this a "minor" problem because 32-bit Mono is just fine from my experience.

So basically if the game engine is written with multiplatforming in mind, it'll work. The language and core libraries don't require Windows.
There are counter-examples of this, like GZDB, which completely depends not only on Windows Forms, but also on DirectX, both of which are not supported properly (and even then, some people claim to actually run GZDB just fine under Wine+Mono).

[Eevee]

Hm. I'd appreciate if we could tone down the vaguely condescending attitude, especially towards beginners (and beginner-friendly languages). This is a video game, for crying out loud. It's a beginner magnet. Frankly, I'd rather see more people create interesting things with bad code than have a smaller community where everything's pristine under the hood. That doesn't mean I'm in favor of PHP-style "anything goes" (certainly not), but "why should we make it easier" is very worrying. We're not writing banking software here; if someone's code sucks, the worst that happens is a few of us frown at it.

Static typing isn't magic, and it doesn't remove the need for tests. Big projects in any language (and especially engines!) have test suites that look pretty similar — because tests are about checking logic, not types. Graf, it's strange to me that you so strongly value having computers check types for you, but you seem ambivalent about having computers check that a program actually works as intended. How can you know ZDoom remains compatible with anything without automated tests proving it? The type system tells you that the program (probably) runs, not that it runs correctly.

The choice is not explicit null checks versus silent failure. I would never propose silent failure; I'm quite strongly opposed to it. (It's one of the reasons I don't much like C!) I'm surprised that you haven't pounced on statically-enforced null checks, when you're so adamant about other kinds of static enforcement. This is a real feature that exists in multiple forms in multiple serious languages.

Please try to step back for just a moment here. You don't see the big deal about nulls, because you're used to languages that have them everywhere, and you've learned to deal with that and spot problems ahead of time. But they're still a huge potential source of runtime errors, and it is possible to minimize those errors, and that should be right up your alley. And conversely, other programmers are used to different kinds of potential runtime errors, and they've learned to deal with those and spot problems ahead of time. It makes no sense to take such a hard line against one kind of issue but be so blasé about a very similar one.

[Graf Zahl]

Hm. I'd appreciate if we could tone down the vaguely condescending attitude, especially towards beginners (and beginner-friendly languages). This is a video game, for crying out loud. It's a beginner magnet. Frankly, I'd rather see more people create interesting things with bad code than have a smaller community where everything's pristine under the hood. That doesn't mean I'm in favor of PHP-style "anything goes" (certainly not), but "why should we make it easier" is very worrying. We're not writing banking software here; if someone's code sucks, the worst that happens is a few of us frown at it.

I have nothing against making things easier. But all the proposals I have seen here just do not convince me. They make it primarily easier to write bad code and that's simply not a good thing. Like dpJudas already said, my biggest issue with Javascript in particular is that the language basically has no self-documenting features, which makes getting into the code extremely difficult. In C++, C# or Java I always have a class declaration which already gives me a lot of info about what the class can do and more importantly, how it manages its data.

Static typing isn't magic, and it doesn't remove the need for tests. Big projects in any language (and especially engines!) have test suites that look pretty similar — because tests are about checking logic, not types. Graf, it's strange to me that you so strongly value having computers check types for you, but you seem ambivalent about having computers check that a program actually works as intended. How can you know ZDoom remains compatible with anything without automated tests proving it? The type system tells you that the program (probably) runs, not that it runs correctly.

Good luck writing unit tests for ZDoom. Sorry, but that goes way beyond my tolerance level. If you look at my own code you'll probably already see an entirely different style that is actually designed to make testing easier. What do you think I use to check the ZScript compiler? The tests I do aren't fully automated, of course, because I constantly need to check the bytecode output to see if everything is ok, but this is broken down in so many smaller parts that it can be properly tested.
But the vast majority of the old code is so complex, complicated and convoluted that all we can do is pray that it never breaks.

The choice is not explicit null checks versus silent failure. I would never propose silent failure; I'm quite strongly opposed to it. (It's one of the reasons I don't much like C!) I'm surprised that you haven't pounced on statically-enforced null checks, when you're so adamant about other kinds of static enforcement. This is a real feature that exists in multiple forms in multiple serious languages.

Well, as a matter of fact you cannot do static checks on data that may or may not be valid at runtime. The best I could imagine is an error telling me that an unvalidated pointer is being used. Of course that would create so many false positives in real-life code that I question its usability.

Please try to step back for just a moment here. You don't see the big deal about nulls, because you're used to languages that have them everywhere, and you've learned to deal with that and spot problems ahead of time. But they're still a huge potential source of runtime errors, and it is possible to minimize those errors, and that should be right up your alley. And conversely, other programmers are used to different kinds of potential runtime errors, and they've learned to deal with those and spot problems ahead of time. It makes no sense to take such a hard line against one kind of issue but be so blasé about a very similar one.

I know myself that for some potential users this may pose a problem, but as things are, they will have to learn that nulls are a very valid and very frequent occurence in this code. I cannot make that magically go away. These pointers can be null, and that inevitably means that code must be aware of this and deal with it appropriately. And no amount of sugarcoating will really do anything good. If modders want to do such low level scripting they have to know how this works or they'll never be able to write working code.

If you have any concrete idea how to improve this, please tell me, but 'nulls are bad' is not enough.

[Major Cooke]

Now I really wish to find out what this is all about. So does that mean virtual functions are temporarily down?

[Eevee]

I have nothing against making things easier. But all the proposals I have seen here just do not convince me. They make it primarily easier to write bad code and that's simply not a good thing. Like dpJudas already said, my biggest issue with Javascript in particular is that the language basically has no self-documenting features, which makes getting into the code extremely difficult. In C++, C# or Java I always have a class declaration which already gives me a lot of info about what the class can do and more importantly, how it manages its data.

At some level, the alternative to writing bad code isn't writing good code — it's writing no code at all. You can help an inexperienced programmer improve, but you can't help someone who gave up before managing to write any working code. I'd rather have a little more bad code than fewer people making things overall.

Speaking as someone who's been having to get a handle of ZDoom guts for the first time recently, I disagree that class declarations are a documentation panacea. I mean, this is a codebase with Actor::special1, Actor::special2, Actor::specialf1, Actor::specialf2, and Actor::weaponspecial. Not that far off from the hypothetical worst-case struct I wrote. I know none of us think ZDoom is the most beautiful codebase to ever grace the earth, but that's exactly my point: static typing doesn't magically make code good.

Good luck writing unit tests for ZDoom. Sorry, but that goes way beyond my tolerance level. If you look at my own code you'll probably already see an entirely different style that is actually designed to make testing easier. What do you think I use to check the ZScript compiler? The tests I do aren't fully automated, of course, because I constantly need to check the bytecode output to see if everything is ok, but this is broken down in so many smaller parts that it can be properly tested.
But the vast majority of the old code is so complex, complicated and convoluted that all we can do is pray that it never breaks.

Forced to choose between the perfect static type system and automated tests, I'd take the tests in a heartbeat. (Hm — I don't know how tests for mods written in ZScript might work, but that's an interesting idea to think about.)

Chocolate Doom has a testing setup that replays demos from Compet-N and verifies that some overall stats at the end came out the same, which is a great way to automate sanity checks. ZDoom demos are more brittle, of course, but you could have test wads that check stuff with ACS and a harness that runs them automatically and reads results from stdout. Tests don't necessarily need to be internal unit tests. I know Firefox has a huge pile of "reftests" that effectively just take a screenshot of a page and compare it to a known good rendering; ZDoom could do the same. Way better than nothing.

Well, as a matter of fact you cannot do static checks on data that may or may not be valid at runtime. The best I could imagine is an error telling me that an unvalidated pointer is being used. Of course that would create so many false positives in real-life code that I question its usability.

It's already done in several real-life languages. I gave examples in Swift and Rust before, but I misread a detail about Swift and I forgot the easiest approach for Rust, so let me try that again.

Rust has no null value. Instead it has a generic wrapper, Option<T>, which might be Some<T> or None. So you can tell right away from type names whether a given value is allowed to be "null": Option<T> is, but T is not. And when T is a pointer type, Option<T> compiles down to a regular nullable pointer, zero overhead.

Code: Select all

// Rust
// let's say foo is inferred to be Option<T> here
let foo = some_func();
// I can't use T methods on foo, because it's not actually a T.
// I can unwrap it with pattern matching, because Option is an enum of Some<T> and None:
match foo {
    Some(x) => ...,  // x is scoped to this expression
    None => ...,
}
// Or for the common case of just caring about the contents...
// ("if let" is a special kind of pattern-matching block, not just expression assignment)
if let Some(x) = foo {
    // This compiles to a null pointer check, and of course only runs
    // if foo contains a value.  x is scoped to this block
}
// Option also has some helper methods.
// For example, this lambda only runs if foo contains a value.
// bar is inferred to be Option<U>, where U is the return type of do_something.
let bar = foo.map(|x| x.do_something());

Rust is 100% statically typed and compiles much the same way as C++, but null pointer issues are greatly constrained. You know exactly which values may or may not exist, and the compiler doesn't let you use an optional value without somehow acknowledging its optionalness first. If you really really want to defer the null check to runtime and accept a panic when it's null, you can do that too — and have a pretty good idea of where the problem is. But completely forgetting to check is impossible. Mozilla is currently writing a browser engine in Rust (and recently started shipping Rust code in Firefox), so it's perfectly usable.

I'm not familiar with the downcasting story, but it'd have to return an Option<T> regardless, so you'd deal with the result the same way.

The especially neat thing is that this is just a regular Rust type, not a magical builtin. ZScript probably doesn't need the type system plumbing that Rust uses to make this work, but it could special-case this: distinguish between nullable and not-nullable pointers, and have some statically-enforced way to only use nullable pointers after a check.

[dpJudas]

Speaking as someone who's been having to get a handle of ZDoom guts for the first time recently, I disagree that class declarations are a documentation panacea. I mean, this is a codebase with Actor::special1, Actor::special2, Actor::specialf1, Actor::specialf2, and Actor::weaponspecial. Not that far off from the hypothetical worst-case struct I wrote. I know none of us think ZDoom is the most beautiful codebase to ever grace the earth, but that's exactly my point: static typing doesn't magically make code good.

Nobody is saying that static typing makes bad code magically good. What it does is making it easier to study and analyze. Both for tools and humans. You quickly listed the 5 most odd things you could spot in an Actor class, but remember, had that been dynamic code you'd not know if there is a special3 as well or not. It also would have been impossible for you to compile that list in the first place without a huge amount of work.

Forced to choose between the perfect static type system and automated tests, I'd take the tests in a heartbeat. (Hm — I don't know how tests for mods written in ZScript might work, but that's an interesting idea to think about.)

There is also nobody here saying that static typing replaces tests. What it does is reducing what needs to be tested. A vector library in a dynamic language needs to be able to handle the user passing in odd stuff like strings. In a static language it does not. The test needs to.. well, test this.

Rust has no null value. Instead it has a generic wrapper, Option<T>, which might be Some<T> or None. So you can tell right away from type names whether a given value is allowed to be "null": Option<T> is, but T is not. And when T is a pointer type, Option<T> compiles down to a regular nullable pointer, zero overhead.

I'm not sure beginners will find it easier to understand they have to prefix with Option<T> or call a strange (to them) target.require("die") function before accessing it. At the end their program crashes the same way - only difference is that it now crashes with a "die" message in the console, while before it would have died with "tried to access a null reference" without this feature.

Don't get me wrong, it is a nice feature for a new language like Rust. I haven't used Rust, so I can't say if this feature would annoy me for being to pedantic or not. If it is worth the effort to implement for ZScript I'm a bit more sceptical about.

[Eevee]

Nobody is saying that static typing makes bad code magically good. What it does is making it easier to study and analyze. Both for tools and humans. You quickly listed the 5 most odd things you could spot in an Actor class, but remember, had that been dynamic code you'd not know if there is a special3 as well or not. It also would have been impossible for you to compile that list in the first place without a huge amount of work.

It sounds a lot like people are saying that, since the argument against dynamic code has largely been "it's easier to write bad code".

Static typing makes for easier static analysis, yes, and a dynamic runtime makes for easier dynamic analysis. It's typical to initialize all your attributes even in a dynamic language, but if I had really goofy code and I wanted to know what an object's attributes ended up being, I could just print them out at runtime. Or I could trivially intercept assignments to them, so I'd know exactly when and where they were assigned. The type is too common and that becomes useless noise? Okay, only trace one particular object, or trace based on an attribute, or trace 5% of the time, or track stats on which attributes are used more often and print out a histogram. I can invent debugging features on the fly, from within the language itself.

I'm aware that static typing has benefits. I'm just saying dynamic typing also has benefits, and they're not all "I get to be sloppy and lazy". I worked on the plumbing of a 1.5+Mloc Python app for years, and I saw plenty of pros and plenty of cons. People did bone-headed things once in a while, but we also had the power to make massive improvements more or less transparently.

There is also nobody here saying that static typing replaces tests. What it does is reducing what needs to be tested. A vector library in a dynamic language needs to be able to handle the user passing in odd stuff like strings. In a static language it does not. The test needs to.. well, test this.

The main dev for a large project with no tests just joked that tests were invented for untyped languages, so...

A vector library in a dynamic language absolutely does not need to handle that. If someone passes in strings, they get a type error the moment they try to do any math. What is there to test?

You might feel inclined to test that your code doesn't pass strings in, but that's a free side effect of having tests in the first place. If the values are fine, the types are definitely fine.

I'm not sure beginners will find it easier to understand they have to prefix with Option<T> or call a strange (to them) target.require("die") function before accessing it. At the end their program crashes the same way - only difference is that it now crashes with a "die" message in the console, while before it would have died with "tried to access a null reference" without this feature.

The primary benefit of Option<T> is that your program doesn't compile in the first place, not that it crashes differently. You can opt into a runtime error, but you have to deliberately ask for it — it's not something you can just forget about, because the compiler won't let you.

[arookas]

I feel like we need some WildWeaselMagic™ because we aren't discussing zscript now so much as we are discussing "dynamic vs static typing". ...and the best way for languages to feature null checks, for some reason?

Hm. I'd appreciate if we could tone down the vaguely condescending attitude, especially towards beginners (and beginner-friendly languages). This is a video game, for crying out loud. It's a beginner magnet. Frankly, I'd rather see more people create interesting things with bad code than have a smaller community where everything's pristine under the hood. That doesn't mean I'm in favor of PHP-style "anything goes" (certainly not), but "why should we make it easier" is very worrying. We're not writing banking software here; if someone's code sucks, the worst that happens is a few of us frown at it.

How are we turning away beginners? DECORATE is easy to get into, learn, and use. Look at how powerful it has become and even half of the amazing things that have been done with it over the years. That isn't going anywhere. For people wanting a little bit more power and flexibility, they will soon have zscript which can be (and will be) its own thing, thankfully. It can be statically typed if the devs feel it is appropriate.

Personally, I feel making zscript statically-typed and OO like it is now is a perfect match for how the engine works internally. The goal of zscript is to externalize the engine's internals, make a lot of things less hardcoded, and make the engine generally more extensible. This seems like the best course of action—putting a dynamically-typed interface over a statically-typed engine doesn't appear to be that intuitive.

For instance, all the amazing and otherwise hidden functions being exposed to zscript are pretty much directly linked to their C++ counterparts, which use statically typed arguments. Imagine how much code bloat it would require for a simple sound function to not be able to expect the sound identifer to be a specific type, or a movement function to have to check if that vector was passed in as a list of floats rather than a single vector object. If the function accepts only one type for a given parameter, why have it dynamic in the first place?

The primary benefit of Option<T> is that your program doesn't compile in the first place, not that it crashes differently. You can opt into a runtime error, but you have to deliberately ask for it — it's not something you can just forget about, because the compiler won't let you.

While compile-time null checks seem pretty snazzy, it doesn't seem that useful for zdoom. Most of your interaction will be through actor pointers and the such, which would need to be nullable (therefore in your case Option<T>'s). This doesn't really solve any problems anymore since it isn't guaranteed to be non-null. As for compile-time null validation, that seems way overboard for something like zscript when everyone else is accustomed to simply checking if it is null before using it. Your "requires" example seems like just another way to type "if (mypointer == null)" but evaluated at compile time. But, feel free once zscript is more full-fledged, ready and shipped to make a feature suggestion (or code submission) for it. I don't see it happening, though.

Besides, adding something like Option<T> would also require zscript to implement generic/templated types, which is something I don't believe it has. For a little bit of syntactic sugar and compile-time safety? Let's keep it simple.

[Eevee]

How are we turning away beginners? DECORATE is easy to get into, learn, and use. Look at how powerful it has become and even half of the amazing things that have been done with it over the years. That isn't going anywhere.

I was referring to Graf's comments about beginner code. But since you bring it up: are we saying that DECORATE is deprecated, and we'll tell beginners to use it instead of the (relatively backwards-compatible) ZScript? That's... inconsistent with how ZDoom deprecation tends to work.

This seems like the best course of action—putting a dynamically-typed interface over a statically-typed engine doesn't appear to be that intuitive.

In fairness, since CPUs are statically typed, every dynamically typed interface is ultimately on top of a statically typed engine.

The little Lua bindings I've been experimenting with use a library that converts between Lua and C++ types completely automatically. I haven't had to write a single check or conversion at the boundary.

As for compile-time null validation, that seems way overboard for something like zscript when everyone else is accustomed to simply checking if it is null before using it.

Are they? This sounds suspiciously like doing it just because C++ does it. Wouldn't someone coming from the rest of ZDoom be more accustomed to nulls (or, similarly, unused TIDs) being silently ignored? (Not that that's good, just saying.)

Besides, adding something like Option<T> would also require zscript to implement generic/templated types, which is something I don't believe it has. For a little bit of syntactic sugar and compile-time safety? Let's keep it simple.

Pretty sure I've seen several instances of Class<Actor> or similar in ZScript code so far, though I don't know exactly how accessible that genericism is to user code.

I would love to keep it simple rather than have a little bit of compile-time safety, but no one seems to agree. I love the irony here — being lukewarm about null checks because they need extra syntax and only avoid occasional problems in practice is exactly how I feel about static typing in general.

[Graf Zahl]

Now I really wish to find out what this is all about. So does that mean virtual functions are temporarily down?

No. The code Leonard made turned out not to be working as expected so I had to redo the entire thing. Actually, virtual functions ARE properly working now, they weren't before.

The only exception is the Destroy method which gets in the way of the garbage collector. If any object with a scripted virtual override table got into the final cleanup action it'd badly crash the engine because there's no guarantee that the needed VM related objects haven't been deleted already.