Found out the actual cause. It's this section of the code. It's not accounting for "out" parameters when calculating the register count.
However, it seems there's still some underlying bug, as the Y component seems to always return 0 for some reason. Not sure what's causing that, though.
This is my proposed fix for the incorrect register count:
// NumArgs for the VMFunction must be the amount of stack elements, which can differ from the amount of logical function arguments if vectors are in the list.
// For the VM a vector is 2 or 3 args, depending on size.
auto funcVariant = item.Func->Variants[0];
for (unsigned int i = 0; i < funcVariant.Proto->ArgumentTypes.Size(); i++)
{
auto argType = funcVariant.Proto->ArgumentTypes[i];
auto argFlags = funcVariant.ArgFlags[i];
if (argFlags & VARF_Out)
{
auto argPointer = NewPointer(argType);
sfunc->NumArgs += argPointer->GetRegCount();
}
else
{
sfunc->NumArgs += argType->GetRegCount();
}
}
Seems to work fine on my end.
Spoiler: Previous, incorrect thread contents
Scratch that, I fucked up when debugging it. Still most likely a bug in the ZScript compiler, though. Will be reusing this thread when I find the actual cause.
Spoiler:
Somehow, it seems vector arguments are losing their flag parameters. This breaks "out" arguments. Thi is what causes this bug.
However, I've noticed Vector arguments ALSO get the VAR_Ref flag by default. If this does what I think it does, that part of the code will also have to be removed once this is fixed so existing mods don't get broken.
Currently working on this, putting it here so I don't forget it.