ZDoom

Basically I need an enemy radar that is rectangular in shape. I'm having some trouble figuring out where to start with this.

Well, I see two ways of going about this:

1. Give literally every enemy ( except those that never appear on the radar, I suppose ) a unique TID, and keep track of the highest TID placed on an enemy so that you can use a for loop to iterate through all of them and get their location.
2. Store stuff like x position, y position, etc. in an array, then make every enemy execute a looping script that assigns them an index in those arrays so you can constantly update those arrays with the values from the actor.

Once you do that, it's mostly just a matter of making sure the location data you're getting is within a bounding box centered around the player ( though you're going to have to deal with some trig if you want the radar to rotate with the player ) and if so, translating the positional offset from the player into a graphical coordinate to use with HudMessage.

That all said, this is obviously some pretty heavy stuff to do with "enemies" meaning "monsters," especially when the monster total climbs up. I've had success with the second method when it comes to making a radar that only works with players, but I'm not sure if ACS is gonna be anywhere near as efficient enough for your purposes.