ZDoom

vid_scalemode 1
vid_scalefactor 1
vid_aspect 0

When you're in fullscreen, type the following commands:
[code]vid_scalemode 1
vid_scalefactor 1
vid_aspect 0[/code]

This should reset the scaling back to normal. If you want to lower the resolution but keep consistency with the font, you either have to enable linear filtering for the whole screen, or use even-size increments for vid_scalefactor (i.e. 1/2, 1/3, 1/4, etc).

If you're using LZDoom though, you also [i]must[/i] use the same resolution as is native to your display in order to be able to control how the font is rendered.

Can it be applied only to the text?
I don't want the entire screen filtered

Just to clarify
In windowed, all resolutions work and don't have the jittery edges
But then in fullscreen it breaks with the scaling

In Options->Set video mode, change "Use Linear Scaling (Fullscreen)" to "Yes".

This only happens in fullscreen and I felt that taking screenshots wouldn't properly detail the issue, so I used a phone.
All three of these are taken in different fullscreen resolutions
[imgur]https://imgur.com/t7NNcPe[/imgur]
[imgur]https://imgur.com/v08j398[/imgur]
[imgur]https://imgur.com/gbgN8Tf[/imgur]
Look closely and you'll see the inconsistancies

Could you be a little more specific? Maybe provide screenshots?

While I like the new font that's been given to GZDoom's menus, it looks kinda janky in certain resolutions under 720p
Can this be remedied?