ReflectionHLE (Reflection Keen)

[NY00123]

[UPDATE: Oct 14 2022] Web page link for reading a bit about the project: https://reflectionhle.com/

[UPDATE: Dec 31 2021] The project is titled ReflectionHLE as of writing this. Ignoring the URL, the rest of this post currently remains unedited by me.

Hi all,

Let me introduce you to the following source ports inspired by Chocolate Doom, now also covering Wolfenstein 3D: https://github.com/ReflectionHLE/ReflectionHLE/releases

This is a 6 years anniversary release. On September 26 2014, I released a new port of Keen Dreams, initially titled "Chocolate Keen Dreams".
Although Keen Dreams was released earlier in the same month, I actually started some work on it even earlier, after the Catacombs were open-sourced (in June 2014). Reason is that I already knew that a lot of the code was common.

Following Keen Dreams, I added Catacomb Abyss, then Catacomb 3-D, and finally the rest of the Catacomb Adventure Series.

Recently, I've been working on the source ports again. With the assistance of the gamesrc-ver-recreation/wolf3d tree, the last release introduces support for Wolfenstein 3D, Spear of Destiny (excluding the mission packs) and Super 3-D Noah's Ark (DOS version).

For Wolfenstein 3D, this currently covers Apogee shareware versions 1.0, 1.1, 1.2 and 1.4 (the one with cheats), as well as 6-episodes Activision v1.4.
For Spear of Destiny, this covers FormGen demo v1.0 and Activision v1.4.
For Super 3-D Noah's Ark, this covers the one Wisdom Tree DOS version that I'm aware of.

There are still some problems. In particular:
- You can't save games or load saved games in Wolf3D and the games based on it.
- These games have no support for the "modern" game controller scheme. I think that the other games should still have it, though.
- For technical reasons, you can't load the Spear of Destiny mission packs.
- More generally, you can't load TCs made to work with original EXEs from the 90s (as released by id and/or related publishers). I still haven't decided how to approach this.
- In order to detect game data, you also need the original DOS exe (albeit the 2015 edition of Keen Dreams is an exception).
- For Super 3-D Noah's Ark, this exe currently has to be named noah3dos.exe, as this is the way it arrives from Steam right now. Currently, there's no good way to support auto-detection of data with alternative file names.
- The rate of palette updates in Wolfenstein 3D is at least somewhat imprecise.
- VSync is disabled by default for now. There are other potential problems with timing in Wolfenstein 3D, which might partially be related to instances in which the game tries to render more often than the host display's refresh rate.
- Stereo panning remains unimplemented.
- For the Wolfenstein 3D based games, if you try to use a wall right after pushing it, the behaviors are essentially undefined.

[NY00123]

I can write that the port has still been updated (mostly for Wolf3D). Today's release introduces saved games support to Wolf3D. They should be written in the same binary formats as the matching DOS versions. On the other hand, due to the nature of the implementation, there may still be bugs.

I also fixed a bug which, unfortunately, may break (partial) compatibility with Keen Dreams and Catacombs saved games. Then again, such compatibility was probably already broken in the same manner with the matching DOS versions.

The preceding release introduced support for the Spear of Destiny Mission Packs, along with other additions and modifications.

Change log for the last release:

Code: Select all

Dec 25, 2020 (v0.32.0):

* KDreams, Catacombs (and Wolf3D): Make it possible
to bind actions to a game controller's D-pad.
* Wolf3D: Support game controllers, in a similar manner to
what's already been covered for Keen Dreams and the Catacombs.
Further add mappings for touch input.
* Wolf3D: Support stereo panning. This might not be fully accurate,
but hopefully, it's not that far.
* Wolf3D: Support saved games. They should be read and written using the same
formats as the original DOS versions. This may currently be buggy by nature.
* SD_TimeCountWaitFromSrc/SD_TimeCountWaitForDest timing fix, applying to
all games, but especially noticeable while playing back demos in Wolf3D.
* Add support for Spear of Destiny FormGen version 1.0, as well as the
variation of FormGen version 1.4 which is actually identified as "V1.4"
in-game. Currently, the one mistakenly identified as "V1.0" won't be detected.
The mission packs' data still needs to be the same as in the
Activision version, thus e.g., including the UAC logs.
* When game installations are shown in the launcher, don't print their
locations for now. This might become more useful later, in case it will
be possible to choose any of multiple game installations which otherwise
match the same version (e.g., due to somewhat differing game data).
It shouldn't matter as much for now.
* A change that may potentially break saved games made
with older versions of the KDreams and Catacombs ports:
The macros COMPAT_OBJ_CONVERT_OBJ_PTR_TO_DOS_PTR and
COMPAT_OBJ_CONVERT_DOS_PTR_TO_OBJ_PTR were fixed. The original sizes
of the object structs, as present in the DOS executables from the 90s,
are now used, instead of sizeof(objtype) from the source ports.
* Other misc. fixes and modifications. Thanks to Blzut3 for
assistance with a subset of the changes in this version.

[NY00123]

Version 0.40.0 has been released. Since Keen Dreams isn't the only supported game, the project is being renamed as of this version. It's now titled "ReflectionHLE".

A side-effect of this is that you'll have to manually migrate saved games and settings. Hopefully, it shouldn't be more than copying a directory or two. See the top of the change log for details.

This version has the experimental addition of keyboard and mouse button overrides. They might not always work as expected. For instance, overridden keys may become unusable for cheat codes.

There's also an APK for Android again. Audio resampling isn't included in it, and there might be other problems related to audio. Generally speaking, the Android builds should be considered less supported.

Code: Select all

Dec 31, 2021 (v0.40.0):

The project is renamed "ReflectionHLE" as of this version.
As a side-effect, users of earlier Reflection Keen versions will have to
manually migrate older settings and saved games. See paths given below.

Windows: %APPDATA%\reflection-keen => %APPDATA%\reflectionhle

macOS: ~/Library/Application Support/reflection-keen
=> ~/Library/Application Support/reflectionhle

Linux - two paths (assuming $XDG_CONFIG_HOME and $XDG_DATA_HOME aren't set):
~/.config/reflection-keen => ~/.config/reflectionhle
~/.local/share/reflection-keen => ~/.local/share/reflectionhle

* The project was renamed from "Reflection Keen" to "ReflectionHLE".
* A single executable can now be used for running the supported games.
It is identified as "ReflectionHLE".
* Due to using one exe, the application icon had its game identifiers removed.
* Additionally, the cfg file was split, with game-specific settings residing
in their own separate cfg files. Some cfg key and value names were further
renamed. Code was added for automatic migration of these from v0.33.1, but
this does not cover the directories which store the cfg files and saved games.
* The command-line options -skipintro and -showslides were removed. Instead,
you can show the intro or slides with sub-gamever selection via -gamever.
* Added the command-line option -listgamevers. It can be used for listing
supported game versions, instead of -?, which doesn't do so anymore.
* There's now experimental support for keyboard and mouse button overrides,
applying to in-game actions during gameplay.
* Note that these take a higher priority than in-game settings for the
same keys. They also have a higher priority than hardcoded behaviors of them.
For instance, it might be impossible to use a cheat code. You can still
use a cheat code via the on-screen debug keys, though.
* The on-screen keyboard with debug keys can now be displayed even
if there's no use of touch input or any game controller.
* Revert the impact of changes from v0.33.0 to the way controller buttons
get mapped under the modern controller scheme. Instead of mapping to the
default settings of a game, mapping is now done to the matching actions.
For instance, in Keen Dreams, you can map a button to Jump instead
of Ctrl (the default key for jumping in the original releases).
This is still done in a different manner from versions preceding 0.33.0,
as it's now less tied to the original games' keyboard settings.
* As a side-effect, there are changes to the impacts of the analog motion
toggles, even while using a D-pad. Additionally, the novert toggle should
also impact vertical motion from a game controller or touch input.
* Catacomb Abyss: Ensure it's possible to scroll through HELP.TXT using a game
controller or touch input, even if movement keys were changed in config.abs.
* General restructuring of the launcher's menus, following the support for
multiple games from a single exe and the addition of key/button overrides.
* Android builds are now possible again, albeit without audio resampling.
Generally speaking, these builds should be considered less supported.
* For technical reasons, sound output is done somewhat differently on Android.
There may still be problems with audio, including delays.
* Improvements to timing of sound playback during the intermission screen
in Wolfenstein 3D, Spear of Destiny and Super 3-D Noah's Ark. This
might be more noticeable while VSync is toggled on.
* Launcher control bindings for Wolfenstein 3D and Super 3-D Noah's Ark:
The weapons/feeders are now referred to by their numbers,
instead of game-specific descriptions.
* Emulation of mouse motion via game controllers was made
less frame rate dependent.
* Fix a typo impacting startup of Catacomb Apocalypse's hint book. The offset
of a textual screen as present in the original exe was wrong. The bug was
introduced during a refactor after the release of v0.30.1.
This screen is actually used just in the electronic catalog of the supported
shareware version of Catacomb Abyss, upon trying to print the catalog.
* Catacombs, Wolf3D: Fix building in case "char" is an unsigned type.
* Additional miscellaneous refactors, edits and fixes.

[NY00123]

Version 0.40.1 has been released. This update mostly consists of usability improvements. Also, for people less familiar with ReflectionHLE, a web page is now up: https://reflectionhle.com/

This release will hopefully resolve usability problems with keyboard overrides, and possibly also with other input devices. But there are great chances that there's a new bug not found by me, so bug reports will be welcome. You can use the GitHub repository's issue tracker, or report in the forums if you prefer so; Albeit I might not necessarily check all forums in which I posted about ReflectionHLE.

Code: Select all

If you are upgrading from a version preceding 0.40.0, please check
migration notes for v0.40.0. The project was renamed to "ReflectionHLE"
at the time, and as a side-effect, older settings and saved games
have to be manually migrated.

Oct 14, 2022 (v0.40.1):

* Add a toggle for showing/hiding the ENDOOM screen on exit,
currently identified in the launcher as "Wait for input on exit".
Regardless of the toggle, it'll still appear in case of a nonzero exit code.
Other screens specific to shareware versions of certain games might
further keep appearing, or introduce their own delays.
* Updated search paths for Wolfenstein 3D and Spear of Destiny game data.
* Keyboard overrides don't fully replace the original uses of the keys
as of this version. For instance, if the launcher is used to select
the key W for moving forward in Catacomb 3-D, it can still be used
as a part of debug keys for warping to another level.
On the other hand, the familiar functionality of the key otherwise used
for the same in-game action will be disabled. To clarify for the example
of moving forward, the key generally used for this purpose from
Catacomb 3-D's own control panel (the Up arrow by default)
will not actually be used for moving forward.
Note that mouse button overrides are still full overrides and there's
no change in behaviors as describe above.
* Miscellaneous internal changes related to user input, related in part to
keyboard overrides. One example is a bug fix, where pressing on a key
moving Keen left, followed by another key moving Keen right, and then
releasing one of the keys, would make Keen stop moving altogether.
Also, as of this version, the analog motion is explicitly reserved for analog
controller axes; novert impacts these, as well as ordinary mouse movement.
* Use more locks for the audio mixer. This particularly
helped resolving unexpected slowdowns in Android builds
with BE_ST_MANAGE_INT_CALLS_SEPARATELY_FROM_AUDIO=1,
during playback of digitized sounds in Wolf3D.
* BE_ST_MANAGE_INT_CALLS_SEPARATELY_FROM_AUDIO=1 is used for Android again.
* Other updates related to Android builds.
* Made a few changes removing dependencies on libgcc and C++ libraries
as long as we don't actually need any of them.
* Other miscellaneous edits.

[NY00123]

Now, there might not be much to add; Mainly a few notes related to known issues with detection of game data in the last October 2022 release, plus a short visit to what was essentially the very beginning of ReflectionHLE.

While ReflectionHLE's first released version, initially titled "Chocolate Keen Dreams", dates September 26 2014 and supports just Keen Dreams, I actually did some early work before Keen Dreams was open-sourced. June 2014 had the open-source release of the Catacomb games (plus Hovertank). Following that, I did some initial work with the Catacomb 3-D codebase, emphasis on code expected to be useful for a Keen game (at least with CGA graphics).

It wasn't possible to make a working executable. It actually occurred more than once while working on a new port of a game, that the process of porting would be done without observing if there's any mistake in runtime before finally getting to the stage of creating an executable. In particular, I don't recall creating a small test game for DOS as a reference.

I had already uploaded a repository with a kind of a (possibly modified) snapshot of such code back in 2014, but as of today, I took 3 old folders and created archives out of them. I also created a new repository, so you may check them out: https://github.com/ReflectionHLE/chocol ... s/releases

Referring to the aforementioned issues in the last release:
- It might fail to auto-detect game data from a GOG.com release of Wolfenstein 3D if it's a "recent" download, which may potentially refer to any time from January 2023 and later (and possibly also earlier). Due to lacking a GOG.com copy of my own, I tried asking a few other people and filled details based on what I learnt. It turned out I used the wrong id somewhere (possibly a store id). It was only by January 2 2023 when I added a commit with the fix. While not fixed in the last 2022 release, you should be able to manually locate game data via ReflectionHLE's launcher.
- If you use Android 11 or later then ReflectionHLE might fail to access compatible game data, even if you enable all permissions that you can. I recall discovering this sometime during last year, after changing to a newer device. Another user reported the problem on GitHub, recently. As suspected by me, that's related to Android's 11 introduction of scoped storage. Since I haven't distributed ReflectionHLE via an online store, I've simply made it depend on a permission to manage storage, usually reserved for specific apps. A test APK can be downloaded, albeit as usual, consider this to be less supported: https://github.com/ReflectionHLE/Reflec ... /issues/20

[NY00123]

So, it should be the 10th anniversary of ReflectionHLE's first release. Hence, there's an updated version now. There's one new feature, at least for Wolfenstein 3D and derived games. Basically, you may load modified game assets! That includes map sets and other modifications that change just game data files. Note that for Spear of Destiny (including the mission packs), ReflectionHLE expects the file extension of .SOD to be used. Further note that certain contents won't necessarily behave the same as with matching DOS versions, or even lead to crashes, due to changing memory in unintended manners. A good example is the map set Temporary Insanity.

This is also an opportunity for writing more here, including the question of the future of ReflectionHLE and other ports inspired by Chocolate Doom.

1. First of all, when it comes to ReflectionHLE and other projects, I've often had a tendency to not make promises before releases. In that sense, there's no significant change here. What I'm adding here is that generally speaking, there are various things that someone might want to do while time is limited, even if there is presumably a lot of free time.

Hence, it might not imply I won't ever add anything to the repository itself, but as far as it comes to ReflectionHLE, consider priorities to be elsewhere. While the following person similarly has other priorities, Blzut3 still has repository access. It was originally granted to Blzut3 in order to publish packages for macOS.

Other people might have a look at the code as they could beforehand; Maybe also work on their own forks. They might find it much more convenient for them to remove the launcher code, and potentially also other UI code (as intended for use with game controllers and touch input).

2. The ReflectionHLE website was an attempt suggested to me at some point. As expected, though, ReflectionHLE doesn't have enough audience that I'm aware of, given what has been covered so far.

3. One more general question is, to what level are Chocolate Doom and inspired source ports useful. After all, it's true that these days, if I try a game originally released for DOS for the first time, I might prefer using an original DOS executable initially (before using a source port). Even Chocolate Doom has its differences from Vanilla Doom v1.9 if one knows where to find them. One good example is the use of a GPL-compatible replacement for the DMX sound library. DOS emulation also makes it easier to play old versions of games, possibly including prototype versions.

And yet, projects like Chocolate Doom still have their uses. Just for some examples:
- Adding quality-of-life improvements that might be more difficult to implement, or less convenient to do, with DOS emulation. Examples are improved support for game controllers and WAD merging.
- Depending on how are things coded, ports like Chocolate Doom might make the computer consume less energy than using a DOS emulator.
- Both of the aforementioned points might be even more important when used with mobile devices and operating systems, operated via multi-touch screens.
- To different extents, the codebases of projects like Chocolate Doom might prove to be useful as a reference for other people, including developers of different source ports. Examples being making it easier to understand code originally being lower-level, as well as making an executable with debug symbols in a way that's more familiar than doing the same with a DOS executable.

4. I'll finish with something which is less about ReflectionHLE as a project, and more about Commander Keen, even if I generally didn't get to work on CK-related projects in the last 4 years (outside of exceptions like ReflectionHLE). Yes, multiple Commander Keen games were reverse-engineered. Yes, entities don't have to make source codes available if there's no contractual or legal obligation, generally speaking.

And yet, I'll still bring past (and present) interest in original sources for Commander Keen 1-6 (or at least, 1-5). The sources are known to exist and, about 10-11 years ago, there was an attempt to get them released. It started with a find of Keen 4-6 sources about 11 years ago, followed up by some work about 6 months later that included code reviewing and figuring out how to make working binaries. Access to Keen 1-3 sources were also gained around that time period. Eventually, though, the release had never occurred, even if there was presumably a kind of an approval earlier.

I don't know if the current situation may lead to a change, but still checking. Other products of at least one related company (think of an old operating system) were open-sourced. So was the game code for the 2023 rerelease of Quake II.

[NY00123]

Since my preceding post has been, well, long, I've thought of adding a summary with a list of shorter sentences/paragraphs, along with a few more details:

1. As stated earlier, the last 10th anniversary release introduced an addition mainly beneficial for Wolfenstein 3D and derived games; Namely, the ability to load game modifications not depending on executable changes, including map sets.

2. Also, less than 2 days since the release, Blzut3 uploaded macOS binaries, available from the usual location on GitHub.

3. I decided to add that, while it does not necessarily mean I won't ever add more to ReflectionHLE, my priorities are essentially elsewhere these days.

4. One thing I hoped doing in the past, was transforming ReflectionHLE into a kind of a general framework for Chocolate Doom inspired ports. In practice, it might not exactly be ready (e.g., no proper API and partially inconsistent function naming), but it should still be possible to implement support for other games if anyone wants to.

5. Of course, ReflectionHLE might still be missing features required for other games that people might want to port. One example is MIDI playback, at least for games not necessarily using their own built-in synthesizers or conversion routines.

6. People might still look at the code, as they could earlier; Maybe also start a fork. Chances are they'll want to remove the launcher code, and maybe also other kinds of UI code.

7. Also, people might get ideas from the ReflectionHLE codebase, but good chances are they'll find it easier to reimplement the ideas differently. Maybe not always, but at least on specific occasions (especially the launcher).

8. For one small example, I defined platform-agnostic integer types, like "id0_long" for 32-bit integers, albeit they're game-specific and not a part of the ReflectionHLE backend. For another example, I made use of gamesrc-ver-recreation/wolf3d in order to support multiple versions of Wolfenstein 3D. However, the way macros added by me (like GAMEVER_WOLFREV) assisted can make code harder to read. Then again, there might be no universal solution for this, and the work should probably be done on a game-by-game basis, either way.

9. As brought up earlier, one question is to what level are source ports like Chocolate Doom relevant when you may play the games via DOS emulation, especially more specific versions of them. The answer is that they still bring their benefits. A list of examples is given in my preceding post.

10. Finally, less about ReflectionHLE and more about Commander Keen: While my interests in general have maybe changed over times, my preceding post reminded past (and present) interest in original sources for Commander Keen games. There's indeed no obligation to release them, but there was a past attempt about 10 years ago that seemed to get close. I don't have an answer to this, but maybe the conditions have changed in the last few years, increasing the chances in case of a recheck.