For now it's solved with a batch file to start windowed. May be the problem is driver related (recent drivers could not deal with the poor ddraw emulation) and is affecting only the doubled and quadrupled low resolution modes not he native ones. I'm thinking about removing those modes for the latest windows versions.
@Graf Zahl: Did the modes strictly higher than 640*480 work properly or this is hopeless?