Restoration of differing games' EXE revisions, Doom engine included

[NY00123]

Hi all,

Let me introduce you to the following collection of repositories: https://bitbucket.org/gamesrc-ver-recreation/

Basically, most of this work involves reverse-engineering of code related to games. However, rather than REing a whole game, what I usually do is RE a different version of an already open-sourced game. The success rates vary greatly, due to the technicalities involved.

I got this idea after the open-source release of not just one, but multiple DOS versions of Softdisk's Keen Dreams title. In particular, I had a look at the revision matching shareware v1.13. Using what I assumed to be the exact process for making the EXE, I got precisely the original EXE from the 90s, byte-by-byte. This process includes usage of the right compiler version, as well as packing the EXE with LZEXE 0.91.

Back to the above-mentioned collection of repositories, a more successful example that you may find is the code for Wolfenstein 3D and Spear of Destiny. With minor exceptions for certain Spear of Destiny EXEs, this should cover EXEs that may be perfectly recreated, byte-by-byte.

Examples are the Apogee versions (shareware/registered versions 1.0, 1.1, 1.2 and 1.4(g) without disabling the cheats), as well as the Activision versions. With the assistance of earlier REing work done by Blzut3, I also added the DOS version of Super 3-D Noah's Ark, which can be perfectly recreated, minus the debugging information. I further have a separate tree for Blake Stone, similarly building upon Blzut3's work for REing Aliens of Gold (the open-source release covers just Planet Strike).

Now, my most recent addition is Hexen 1.0, with version 1.1 also being covered. You can find these under the "hexen" submodule.

In the case of Hexen, especially version 1.0, the EXE isn't exactly matching in layout, although there are great chances that it's otherwise identical in behaviors. Saved games compatibility might be an exception; I'm not sufficiently familiar with Hexen's saved game format to know for sure.

More notes to add:
- Like the original open-source release of Heretic and Hexen, this is not including the proprietary DMX sound library. I've added a GPL-compatible replacement, albeit it won't sound exactly the same, especially the music (excluding CD Audio). Some of you may recall Nuke.YKT's PCDoom port. As a part of his work on it, he created a DOS-compatible DMX wrapper which is backed by the Apogee Sound System. So, I've made a few changes to the wrapper and then uploaded it. You'll need to have the submodule subdirs of "audiolib", "apodmx" and "hexen" residing in the same directory.
- As usual with public repositories, there might be changes later, like using output directory names differing from "10" or "11". Just for one example of an explanation, I've recently learnt that there might actually be two original Hexen DOS EXEs identified as 1.1, with one of them possibly including the following A_SoAExplode bugfix for DK, which I've disabled for now: https://bitbucket.org/gamesrc-ver-recre ... lines-1161
- What about other id Tech 1 games, like Doom or Heretic? Well, Doom is expected to be quite more difficult, since we don't have access to original DOS sources. Heretic should hopefully be closer to Hexen, in case I get to it; Although, while working on Hexen 1.0, I occasionally used Heretic code as a reference (even bringing a few functions from Heretic as-is). I might not have this privilege for older versions of Heretic. What I can say, though, is that an earlier inspection of Nuke.YKT and me shows that the Heretic sources appear to match version 1.3.

[Blzut3]

I've recently learnt that there might actually be two original Hexen DOS EXEs identified as 1.1

Given the name of the macro, are you speculating that this second 1.1 exists or has it been found in the wild?

[NY00123]

Given the name of the macro, are you speculating that this second 1.1 exists or has it been found in the wild?

The usage of this macro was added near the beginning, and was technically the first usage of any such macro in the git log for Hexen. So, I just used AV_HR_OPENSOURCEREV, in a similar manner to the Duke3D tree having AV_DR_DN3DGPLSRC for the "source" dir of the GPLed release from 2003 (excluding changes which were probably done by Charlie Wiederhold before releasing the code).

I found an alternative 1.1 EXE, which differs from the one available from Steam, in this repository: https://github.com/Doom-Utils/iwad-patc ... la-engine/

A short while after my original post here, I verified that this other EXE is indeed matching the Hexen sources as originally released. In particular, these are the few differences that I could find:
- The A_SoAExplode bugfix being present in the EXE from the above repo (not fixed in the EXE from Steam).
- VERSION_ID is set to CBI in the earlier 1.1 EXE from Steam, and to BCP in the later one.
- The expanded __DATE__ string differing as expected.

[NY00123]

The Hexen repository has been updated.

- First of all, albeit this dates back to the end of last May, the repository should now cover two variations of version 1.0, as well as two variations of version 1.1.
- As previously stated, the two EXEs identified as 1.1 differ just by the A_SoAExplode -nomonsters bug fix and the VERSION_ID string.
- Regarding the two EXEs identified as 1.0, they differ just by making the call to S_StartSongName from P_SetupLevel conditional, depending on the value of i_CDMusic.
- Finally, a major addition to the repository is Shareware Demo v1.0.

Due to the lack of a preprocessor macro for the demo, I had to manually figure out which definitions are missing. At the least, there wasn't a lot of new code to add.
Examples of differences:
- Various sprite, state and map object definitions are omitted.
- A_ACTION.C:A_SoAExplode is not present in the demo.
- SB_BAR.C: The cheat codes differ, and so does their encryption.
- R_DATA.C:R_InitTextures: ST_Progress is called once per 8 textures in the demo, instead of once per 32 textures.
- H2_MAIN.C: It looks like the DoTimeBomb function was repurposed for the demo.
- P_SETUP.C: DEFAULT_SKY_NAME was changed from "SKY1" to "SKY2" for the demo.
- A_SmokePuffEntry and A_SmokePuffExit were (probably) moved to the end of P_ENEMY.C in the demo version. As a side-note, A_SmokePuffEntry itself isn't present in v1.1, and there's currently no use of it in the codebase at all.
- I_IBM_A.ASM:I_ReadJoystick_: pushad and popad were replaced with pusha and popa, respectively.
- HEX.LNK, HEX_DMO.LNK: i_cdmus.obj is the first obj file to be listed for the demo.
- P_ENEMY.C:A_Explode: For an actor of type MT_HAMMER_MISSILE, damage isn't explicitly set to 128 again in the demo. For MT_ZXMAS_TREE, distance is set to 40 in the demo, instead of 64.
- P_ENEMY.C:A_DemonAttack2: P_SpawnMissile is called with a hardcoded mobj type of MT_DEMONFX1 (MT_DEMON2FX1 isn't present in the demo).
- P_SPEC.C:P_ExecuteLineSpecial: If the input special value is 74 (Teleport_NewMap) and all conditions required for calling G_Completed in non-demo versions hold, then G_Completed may be called only if args[0] <= 4. In case args[0] > 4 and &players[consoleplayer] == mo->player, an "ACCESS DENIED -- DEMO" message is prepared to be shown.

[NY00123]

Hi all,

I'll first add a little reminder that what I did with Hexen can't be easily done with Doom; Reason being that we got original DOS sources for Hexen (which turned out to match a later "v1.1" variant), while for Doom, we got linuxdoom-1.10.

On the other hand, I've added a new repository for Heretic. It may currently be found in the same location as before (i.e., where apodmx and hexen are present).

As I wrote earlier, the open-source release of Heretic essentially matches version 1.3. What I did was recreating the code for version 1.2, which isn't very different from 1.3, and then also for shareware and registered v1.0. The latter two EXEs are again similar to each other. The most significant differences are between versions 1.0 and 1.2/1.3.

I also updated the apodmx repository, so you can now generate a library which is compatible with the Heretic sources as originally released. All I did was replacing the _dpmi_lockregion and _dpmi_unlockregion macros with equivalent functions.

Regarding Heretic, as of writing this, you can still find in P_PSPR.C, INFO.H and INFO.C recreated Phoenix functionality (including an mobj type) that I'm not sure what was its purpose. It's actually possible that it's unused in v1.0, thus explaining its later removal.

Another unused function in which ones may have interest, is the one which I called A_Flash for now. It's very similar in structure to A_Raise, and is the only one making use of weaponinfo_t's "flashstate" field. In practice, this field never differs from NULL in v1.0, so it's probably just another case of incomplete or removed functionality.

My setup of the scripts for the modified Heretic source tree may seem a little bit more convoluted. It basically turned out that the output of Watcom C, at least with version 9.5a and/or close enough, can be impacted by setting an environment variable which should otherwise not be related at all. Even without that, there's still the problem where the compiler inserts environment and/or source code dependent data between C string literals.

On the other hand, we're more lucky than in the case of Hexen v1.0 in one important manner: Assuming DMX is used, I could get the global variables to be ordered exactly as with the original EXEs. This means that their addresses matched the originals. Additionally, at least in my case, each generated binary wouldn't differ from the matching original one (minus the DOS/4GW loader) by more than 600 bytes.

[NY00123]

Hey there,

I've gotten another addition for the Hexen repository. To make this short, it should now cover code which is more-or-less fully equivalent in behaviors to the 4-level beta (Oct 2 1995).

There's probably too much to write about the code itself, so I'll simply mention the following examples of information:
- This revision has code for the removed fly creature (https://doomwiki.org/wiki/Fly).
- The 4-level demo from Oct 18 1995, previously named HEXDMO10 in my repository, was renamed HEXDM10B, while the earlier demo was named HEXDM10A.
- I considered renaming HEXDM10B back to HEXDMO10, while HEXDM10A would be renamed using (a subset of) the EXE's original modification date; Reason being, the latter is identified as a beta in-game, and I already did a similar thing with a Wolf3D proto. beforehand.
- However, both versions are referred to as demos in the README.TXT files from 1995. I also don't currently recall any mention of a version number, like 1.0. For now, I just keep using the names of HEXDM10A and HEXDM10B.
- I originally started to inspect the 4-level beta as a possibility after finishing with Heretic 1.0-1.2. I eventually returned to the beta more recently. What's clear is that it required more work to recreate the code than the previously covered versions of Hexen; Maybe even more than all previously covered versions of Heretic and Hexen, combined.
- In addition to the previously known issue of global variables not being fully ordered as in the original EXE, there are also a few functions that I couldn't get their compiler-generated layouts to fully match. Unless I missed anything, they should still match in behaviors. The functions in question are A_Quake, P_XYMovement and P_ZMovement. The latter's C code was actually not changed by me at all.

[Gez]

nitr8's wish, finally fullfilled!

Do you confirm Randi's findings?

Code: Select all

    // The version from the retail beta is not so great for general use:
    // 1. Pick one of the first fifty thinkers at random.
    // 2. Starting from that thinker, find one that is an actor, not itself,
    //    and within sight. Give up after 100 sequential thinkers.
    // It's effectively useless if there are more than 150 thinkers on a map.

[NY00123]

nitr8's wish, finally fullfilled!

I guess that's the closest to it!

Do you confirm Randi's findings?

While I reversed the behaviors from the 4-level beta (rather than the retail store beta), randi's description looks quite close.

The fly actions themselves may currently be found here: https://bitbucket.org/gamesrc-ver-recre ... #lines-615

The fly's states in the state table are here: https://bitbucket.org/gamesrc-ver-recre ... lines-2874

The same file also has an mobjinfo struct, which may currently be found by looking for MT_FLY. As expected, its DoomEd number is 112.

[NY00123]

So, it's possible that anybody who tries to cover all versions from a single EXE might find it difficult to do so at this point.
That is, unless you go for what I did in Reflection Wolfenstein 3D, as it more-or-less covers separate builds of different game versions in a single exe.

As of this post, the following additions are now in:
- 3-level shareware beta of Heretic (Dec 20 1994).
- Retail store beta of Hexen (Sep 26 1995).

These will probably be the last additions related to Heretic and Hexen. As usual, there's no promise regarding any future endeavor.
To summarize, we ended with 5 distinct DOS builds of Heretic, and 7 of Hexen.

As I suspected, it still took significantly more time to work on the beta release of the 4-level Hexen demo (given preceding work on v1.0 and the demo re-release), compared to the work on the aforementioned two betas of Heretic and Hexen.

In certain technical manners, the two Hexen betas might be close, even if obviously not identical. The aforementioned removed fly code is the same, just for one example.

I did have issues with the generated layouts of the following functions in the Hexen betas again: A_Quake, P_XYMovement and P_ZMovement. As mentioned earlier, the latter's C code was actually not changed by me at all.

In the case of Heretic, I suddenly spotted a different layout for M_FindResponseFile in a recent build of 1.2. I did it see beforehand, also in Hexen, but I thought that it was not reproduced after I finished working on each of the differing Heretic versions, excluding the beta. At the least, the code size, including padding for alignment, remained the same, so the other functions did match in layouts and locations.

[nukeykt]

Hi,

As you know Doom source code as released in 1997 was not for original DOS version, but rather was Linux version cleaned up by Bernd Kreimeier. During clean up, code specific for DOS version was stripped out. Another big change was reorganizing header files, specifically doomdef.h, p_local.h and r_local.h were split to smaller header files. This is where Heretic sources comes in handy. Raven released their code as is, without any sort of clean up. Specifically it has majority of Doom's DOS specific code and header files organization is very close to Doom's original form.

The goal of this project is to combine both these sources and get codebase that is close to the Doom's original codebase as possible. Actually I had this idea for a pretty long time and had previous attempts in the form of PCDoom and PCDoom-v2. This time I decided to start mostly from Heretic sources, gradually reverting Raven's code changes by comparing it to Linux Doom sources. I targeted id Anthology Final Doom EXE as it was closest to linuxdoom sources. Eventually I got compiling EXE, but as expected it had lots of differences to original EXE files. Then I gradually fixed all the differences and got EXE that identical to original EXE by behaviours. Once I covered both revisions of Final Doom, NY00123 volunteered and also covered a lot more Doom revisions down to prototype v1.666. This also includes Chex Quest.

Meanwhile I decided to try to do the same for Strife executable using restored Doom code. As Strife source code was never released, I had to rely on reverse engineering of original EXE file. After finishing initial reversing resulted EXE had some differences to original EXE. Eventually after lots of trial and error and not without some help from NY00123 we eventually got EXE that identical to original EXE (up to usual garbage data between string literals and differences due to __LINE__ macro).

As original release of Doom this does not include proprietary DMX code. You can use APODMX replacement library instead to get complete GPLv2 compatible codebase.



TL;DR These repositories recreate different revisions of Doom and Strife codebase. Compiled EXE files are identical to original EXE in term of behaviours. In total both repositories cover 15 different revisions of Doom, Strife and Chex Quest.



List of covered Doom revisions:

• Doom II prototype v1.666
  Doom Shareware / Doom II early v1.666
  Doom v1.666
  Doom II v1.7
  Doom II v1.7a
  Doom II French v1.8
  Doom / Doom II v1.8
  Doom / Doom II v1.9
  Doom v1.9 Special Edition prototype exe
  The Ultimate Doom
  Final Doom
  Final Doom, later id Anthology revision
  Chex Quest

List of covered Strife revisions:

• Strife v1.3
  Strife v1.31

Doom repository
Strife repository

[Rachael]

That is amazing how you got the source code release to match the original various versions of the .exe files down to such minute details. Good work.

[NY00123]

That is amazing how you got the source code release to match the original various versions of the .exe files down to such minute details. Good work.

Thanks for your interest and support altogether!

For the sake of completeness, I'll mention additional (and partially less Doom-related) updates to gamesrc-ver-recreation, available before or during the same day the Doom and Strife trees were published. Additional notes specific to the Doom and Strife restorations (probably more for Doom) might be left for a separate post.

- Nuke.YKT updated the apodmx repository, i.e., his DMX wrapper that uses the Apogee Sound System. This technically impacts not just Doom or Strife, but also Heretic and Hexen.
- Ken Silverman figured out how was Ken-Build's GAME.EXE (probably) built from the sources as uploaded in November 2002. So, while the code was there beforehand, I added it as an option to the BAT files and make file.
- For a few exes that can be built from the duke3d tree, 3 bytes would differ before adding these linker directives: "segment type code lo", "segment type data lo". As it recently turned out, an alternative fix was the simple removal of the directive "system dos4g".

Additionally, Nuke.YKT is now a part of gamesrc-ver-recreation.

There might still be restrictions on what's uploaded to gamesrc-ver-recreation. For instance, a reversed engineered game is generally not covered. Exceptions are still a possibility. For instance, after Blake Stone: Planet Strike was open-sourced by Apogee, it was stated that the Aliens of Gold sources were assumed to be lost, thus explaining their lack. Therefore, I didn't mind building upon Blzut3's earlier reverse-engineering efforts, and later uploading reconstructed sources for the game.

[nukeykt]

Update to the Strife restoration

A couple more revisions of the Strife executable are covered now: registered v1.1(aka v1.0) and registered v1.2. Both reconstructed EXEs are identical to the original EXE files (up to garbage data between string literals and differences due to the __LINE__ macro). Thus gamesrc-ver-recreation now covers all known registered versions of Strife.

The next obvious step is to try to cover the demo versions of the Strife, but I expect much more differences because both demo versions use much earlier revisions of the executable, so I guess I'll leave this for later.

[Redneckerz]

Update to the Strife restoration

A couple more revisions of the Strife executable are covered now: registered v1.1(aka v1.0) and registered v1.2. Both reconstructed EXEs are identical to the original EXE files (up to garbage data between string literals and differences due to the __LINE__ macro). Thus gamesrc-ver-recreation now covers all known registered versions of Strife.

The next obvious step is to try to cover the demo versions of the Strife, but I expect much more differences because both demo versions use much earlier revisions of the executable, so I guess I'll leave this for later.

You know what i think of it, and you know that there are projects using your source base So thank you for doing the impossible and making it possible.

[NY00123]

To quote Nuke.YKT and myself from about a week ago, before adding a more recent small technical update of the last hour:

To celebrate upcoming doom's 30th anniversary gamesrc-ver-recreation now covers original shareware v0.99/v1.0 release

https://bitbucket.org/gamesrc-ver-recreation/doom

And so, Nuke.YKT decided to go even further than version 1.2, all the way back to the very first proper shareware release! On a side-note, so far, outside of Chex Quest, all covered versions included clear support for Doom II, to different extents. The sprite and mobj definitions were also the same. With shareware v0.99/1.0, the situation is obviously different now.

By the way, I've had really minor updates to batch scripts in the Doom, Strife, Build and Duke3D sub-modules. Earlier in this November, it was discovered that the lack of use of the command-line switch /S for the program CHOICE was apparently a mistake. It might not be observed as a problem in DOSBox v0.74-3, but it otherwise was. Let's hope this is fixed now.

Now, after bringing up the topic of running a reconstructed Doom v0.99/1.0 binary that uses APODMX, Nuke.YKT realized that APODMX was incompatible with v0.99/1.0. That was correct, due to API changes in the functions SFX_PlayPatch and SFX_SetOrigin: https://doomwiki.org/wiki/Random_sound_pitch_removed

Thus, I decided to update the APODMX repository, so it should let you build two versions using the different APIs in separate sub-folders now. I also updated all of the Doom, Heretic, Hexen and Strife repositories accordingly. I further applied technical modifications to the Heretic and Hexen repositories related to the DOBUILD.BAT and DOCLEAN.BAT scripts, making them and related files closer to the ones for Doom and Strife (and partially also Build and Duke3D).