ZDoom

Now it's fixed. Interestingly, in the original game these would have registered as misses entirely.

I finally found out what's at play here:

_mental_'s analysis only found the trigger condition for the misbehavior but not the real cause. What happens here is that the target gets hit by the attack from above, not from the side.
P_LineAttack deals with this case, but P_AimLineAttack does not. So the target scan fails but the real attack hits, and none of the melee functions can deal with this case correctly.

With autoaim on this case just got masked by the wider aiming range that made parts of the target getting covered.

This will require some more thorough checks in P_AimLineAttack, so that it also finds targets getting hit from above or below.

4 years old, eh? I haven't used autoaim for longer than that. I'm surprised I never noticed it sooner. At first I thought it had something to do with my weapon decorate code. I tried to track down the problem, but it resisted all attempts. That's when I began to suspect that it was an actual ZDoom bug.

I managed to track it down to a particular change. This was introduced in r4253, this line to be precise.
Before that commit presence of DF2_NOAUTOAIM in dmflags2 was ignored in P_AimLineAttack() function. But now with autoaim turned off vrange will be always zero.
I don't know was it intentional or not but something close to old behavior can restored by userinfo_t::GetAutoaim() function (BTW clamping will be no longer needed) instead of GetAimDist().

Here is a video of the bug. I also noticed that for the times the ammo isn't used, you don't have that sudden jerk that you normally would have.

I've noticed that attacks such as A_CustomPunch and FAxeAttack won't always use ammo. After some testing, I discovered that if you hit something at an odd angle, it will do the damage and spawn the puff, but it might not use up the ammo.

In this example map for Hexen, I have modified a pot to have a ton of health and make it non-pushable, and I gave the fighter's axe an alt-fire which uses CustomPunch instead of FAxeAttack, but otherwise it does the same thing. First, turn off autoaim (with autoaim on, hitting the pot will always use mana). Then get right up to the pot and aim at the very top of it with either fire mode and even though it will hit the pot, no mana is used.