If an actor uses sprites of a different name in its different states, Zdoom will use the sprite "family" from the see state for the cast call even if the missile and death states use a different sprite name.
ACTOR TestGuy
{
SeeSound "grunt/sight"
DeathSound "grunt/death"
States
{
Spawn:
SPOS AB 10
Loop
See:
POSS AABBCCDD 4
Loop
Missile:
CPOS E 10
CPOS F 8
CPOS E 8
Goto See
Death:
TROO I 5
TROO J 5 A_Scream
TROO K 5 A_Fall
TROO L 5
TROO M -1
Stop
}
}
The actor "testguy" is absolutely minimal, having just enough to show the problem. It happens on genuine enemies too though. You can see that he uses SPOSS sprites in his spawn state, POSS sprites in his see state, CPOS sprites in his missile state and TROO sprites in his death state. However, in the cast call, the game tried to use POSS sprite throughout. This goes most obviously wrong in the death phase where the TROO sprite frames are not a direct mapping to the POSS ones so it looks odd.
I've attached the above files in a zip for quick testing.
Interestingly, this limitation existed back in vanilla, too. If you uses DeHackEd to change the sprite names for a frame or two on an actor, it would stick to the name in the first See state.
Not that the limit should still stick around, mind you. I'd say it's necessary for the possibility that a monster might exist which exceeds the normal maximum number of frames.