I doubt this would need particularly complex "broken map fixing". For most cases, throwing away individual unconnected lines and "welding" the vertices of technically-not-closed shapes should be more than enough.
For the rest of the cases, it can connect the open ends of the lines and hope for the best (chances are the issues will be minimal in 90% of maps, given what it'd be used for), and provide a way for adding "fixes" to maps externally for the rare case this isn't enough.
Those that need manual fixes would honestly be a lost cause for automatic fixing, anyway, and in my opinion simply aren't worth the effort and the risk of breaking other maps trying to fix them. Acting like whatever is done needs to be perfect is counterproductive- there's no way to fix every single case correctly without external input.
In my engine, I triangulate sectors planes as such: (This should be helpful to anyone triangulating Doom maps)
- Trace the individual shapes by walking through the sector's lines, starting at some arbitrary line, it doesn't matter which.
- Start with some line. Use V2 as your current vertex and V1 as your previous vertex. [1]
- Iterate through V2's attached lines.
- Ignore the current line, when you reach it. [2]
- Ignore any lines we've already seen while tracing the shape.
- Ignore any lines that don't have our sector as one of their sides.
- Get the opposite vertex from V2, your current vertex. You'll have to check whether the current vertex is V1 or V2 of the new line, and grab the opposite of that. This will be "V3", your next vertex.
- Calculate the angle made by V1, V2 and V3.
- Select the line pair with the smallest angle, and now V2 will become V1, V3 will become V2, and we'll go back to the beginning.
- We've found a shape when we get back to our first vertex. If we end up at a vertex with no other lines, we've found an unclosed shape.
- Build a "shape tree" from these shapes. If a shape is entirely contained by another, it's a child of that shape. (I use the ray casting algorithm)
- Immediately flatten the shape tree, for simplicity.
- Shapes with even depth are solid shapes.
- Shapes with odd depth are holes in their parent shape.
- Shapes with a hole as their parent are solid shapes inside the hole, and can be simply turned into a depth 0/root shape, flattening the tree.
- Cut holes in the shapes. [3]
- Triangulate by ear-clipping. [3]
- (Optional - I currently don't do this) Improve the triangulation by performing a Delaunay triangulation on your triangulated shapes.
Notes:
[1] You'll find your unclosed shapes during this step. Probably best to deal with them at this point.
[2] You can also ignore lines that have both sides pointing to your current sector, depending on what they are. Since this is for a navmesh, you'll probably have to take into account whether they're blocking, though.
[3] As described in the paper
Triangulation by Ear Clipping, by David Eberly.