ZDoom

I have a sector with 5 floors, each floor has a monster, the bottom one works fine, but the others just stand still and take damage without retaliating.
I've attached an example WAD file to the post so you can get a better idea of what I'm dealing with.

Without being able to view the map right now, the topic "inside" 3D floors warrants the question: Are you really putting the monsters *inside* the 3D floor? That won't work. The inside of a solid 3D floor is solid so any monster being placed there is totally blocked from moving.

Yes, on a quick check, that does seem to be exactly what the map is doing.

@Koris, you'll have to rethink how to do this area: perhaps make individual solid thin floors for the enemies to stand on and, if you still need to use 3D floors to draw the bars on the sides, put additional 3D floors between the actual solid floors, make these 3D floors non-solid and leave the lines as impassable so that the bars still seem to block movement.

But are the floors themselves placed incorrectly? Or just the monsters?

A bit of both. If you want things to be able to move on those platforms, then only the platforms should be solid, not the air space between floors. If you want monsters to move they should not be placed inside solid 3d floors.

1. Monsters can't move within 3D sectors.
2. You should make thin floors as your 3D sectors.
3. Here is your map redone as I would do it. Although, you better run when they see you.
4. I made the bar 3D sectors non-solid so everyone can shoot through them, but made the outer lines block player and monsters, so you can't go in and they can't come out.
5. Another thing, your vertices aren't aligned. If you zoom in you'll see multiple vertices real close to each other. Not good.

Edit: My response was prior to me knowing about the 3D Floor Mode in UDB. Used it for a few hours. Same things apply. You just need to make thin 3D Floors for the Monsters to stand on. If you place Monsters within the thicker 3D floors you made then they are stuck in there like it's invisible concrete. Still need to make the outer railing sectors non-solid or they will never see the Player. Make the outer ring of lines block Player and Monsters.