GZDoom localization development thread [Split]

[Kostov]

One question: I noticed that your fonts only cover the character range from 0x410-0x44f, but Cyrillic also uses 0x400-0x40f and 0x450-0x45f.
What languages are using these additional characters?

Those are languages like Kazakh and Ukrainian. I’m not too familiar with any Cyrillic-using languages besides Serbian and Russian. (The Russian alphabet also fully covers Bulgarian, which is a nice plus. Maybe we should keep a list of all supported alphabets somewhere?)

[Graf Zahl]

It's not that easy. The console font fully covers Western European (ISO-8859-1 and Windows 1252 extensions) and once your stuff gets added Russian Cyrillic. It's still missing extended Latin (U-0100-U-17f plus a handful of extensions.) Most character ranges are too complex for full support (don't expect me to invest time in right-to-left scripts like Arabic or Hebrew - or CJK, for example) or even for all the Latin code points, outside the range up to 0x17f it's mostly too esoteric - I think aside from 4 Romanian letters it's only for languages that have no practical use.

For the other fonts it looks a bit worse. The small font has most Western European characters but not all and the big font has none of these so far. For the other games things look even worse. Hexen and Heretic only have rudimentary additions, Chex Quest has none, and don't even start with other IWADs like FreeDoom or Hacx.

I think the only way to get better coverage here later is that once the Unicode support is more complete trying to crowdsource this, along with new translations.
This will also need some rework of the string table code, which currently suffers from a few design flaws, e.g. if a mod replaces some English text, this won't affect any of the other languages.

[Kostov]

Maybe Gez, who I know to have knowledge about at least a couple of Romance languages, would be willing to help with creating new characters? With dedication, I don’t think doing them will be that hard. Just make different accents and copy them above various letters that already exist, and then we’ll have covered most letters in Windows-1252.

Some people have suggested switching absolutely everything to TrueType. Though, I personally don’t like that option.

[Graf Zahl]

TrueType would be ok for the console and probably also for the option menus which suffer a lot from the inconsistent font sizes in various mods, and even more from some of the fonts not being particularly well suited for display on a screen full of text. That latest one which irked me is Eviternity. It's perfectly fine for displaying some short in-game messages, but having to operate the menus with it is, to be blunt, a major pain in the ass.

The in-game notification may also work, but that has to be an option.
I wouldn't want to touch anything else, certainly not the main menu or script output.
The biggest problem with TrueType fonts is Linux. I already have code for this for both Windows and macOS from an old commercial game project I worked on, but on Linux the situation is a mess. Like almost everything on Linux, this isn't the system's business, you need some third party libraries to do the job and most of these flat out suck at composing text. So I'll probably do the Windows/Mac implementation and leave Linux to somebody else, which most likely means that nothing will happen...

[Graf Zahl]

There’s one gripe from previous versions that I haven’t mentioned earlier, and I don’t know whether the addition of Unicode has fixed it: player names written in Cyrillic don’t save after GZDoom is closed. Instead, the player name is blank when restarting the game.

Just checked, this is caused by the INI parser being a pile of garbage. It contains quite a bit of code that feeds signed chars to the is*** ctype functions which makes them throw up if UTF8 (or even ISO-8859) gets stored in the INI. Extended characters get interpreted as negative and thus considered whitespace which then gets removed.

I don't know if I can fix that, but this may be the final nail in its coffin if the entire thing is suffering from these problems. The only reason why I haven't migrated the config to JSON is that some people consider JSON less editing friendly, but this may be a good reason to finally do it anyway and leave the INI only for the autoload sections.

Of course the person who decided long in the past that chars are to be signed should rot in hell and suffer eternal damnation. This is the single most aggravating design idiocy in the history of C which never got fixed.

[Gez]

One question: I noticed that your fonts only cover the character range from 0x410-0x44f, but Cyrillic also uses 0x400-0x40f and 0x450-0x45f.
What languages are using these additional characters?

Wikipedia has a list:
https://en.wikipedia.org/wiki/Cyrillic_ ... in_Unicode

There's a lot of symbols that are extremely unlikely to ever be needed in GZDoom.

[Kostov]

I’m actually working on making BIGFONT characters for (nearly) all Slavic Cyrillic characters right now, for Doom. I’ll send a pull request once they’re done.

I wonder, which one of these three characters looks the best?

[Graf Zahl]

The last one is a bit too dark. I prefer the first one.

[Graf Zahl]

I just committed the console font rework. Unlike the other fonts I kept the sheet format here because with all characters having the same size there is no point taking them apart. But there's no need compiling them anymore, they now get used directly as images.

Right now, giving a map a Russian name prints it correctly in all places - the intermission screen with BigFont, the automap with SmallFont and the console with ConsoleFont.
I think the bulk of the work is done now. Next I want to fill in the BigFont's gaps in the Latin Supplement range (0xc0-0xff) and the console font in the Latin Extended A range.

[Kostov]

Now that's great news! If you want to make language switching easier for new users in the future, here's a new pull request.

By the way, I just dusted off a WIP Serbian translation I worked on with a friend a while ago. Just converted it to the new format. Perhaps I can submit it to GZDoom before the next release.

I'll also try to make at least a few European bigfont characters tomorrow. The Cyrillic ones are done.

By the way, how does the new console font system work? I see there are two images, one for the Windows-1252 encoding and one with the Cyrillic images. How does the engine know exactly which characters to replace with those Cyrillic ones?

[Graf Zahl]

The files need to be named after the first character in a set, plain and simple.

I added the Latin extended sheet for 0x100-0x17f for the console font already because it was mainly simple accent edits, so it was essentially less work to do it completely rather than investigate the individual usefulness of some characters.

[Graf Zahl]

I'll have to check the language menu to see if this requires a menu restart for the layout. That's the main reason why it never got turned into an option.

[Kostov]

I added the Latin extended sheet for 0x100-0x17f for the console font already because it was mainly simple accent edits, so it was essentially less work to do it completely rather than investigate the individual usefulness of some characters.

Great. I’ve already finished Æ and Þ for the BIGFONT in both their forms. I’ll continue work tomorrow.

This is what the Cyrillic characters look like in their current form. I might redo some of them, but the blueprint is done:

I'll have to check the language menu to see if this requires a menu restart for the layout. That's the main reason why it never got turned into an option.

The user will need to exit out of the options menu in order for the layout to center itself properly, but other than that, everything is instantaneous.

[Kostov]

Jimmy helped me make accents for all European characters just now. We might get the entire block done today!

[Graf Zahl]

Nice!

That leaves the biggest gotcha in the whole scenario: The text-containing graphics. The thing here is: I have no good idea how to deal with this that won't cause any problems down the line.

Solution 1: Require replacement graphics and implement a texture replacement scheme based on the current language
Pro: Only requires work on texture management, the drawing code can be left as-is.
Con 1: Forces creation of excessive assets.
Con 2: suffers from texture prioritization issues because the texture manager needs to decide whether a localized alternative should be used or not. The main problem here is that no other relevant Doom port even has the slightest bit of localization features, so every non-GZDoom exclusive mod comes with their own set of menu and level summary screen graphics, mostly using fonts that do not exist in raw form usable for generating graphics from text.

Solution 2: Same as 1, but allow ad-hoc creation by letting the font print into a texture.
Pro: This solves Con 1 of the above
Con: This doesn't solve Con 2.

Solution 3: Replace everything with text
Pro: Removes the need to provide new graphics
Con 1: A lot of code needs to be reworked to implement this, especially selection logic to decide whether to use graphics or text variants.
Con 2: Creates both visual and technical disparity between languages, so overall not really feasible. I think this can be used for one thing only, namely to replace the (C)WILV graphics, but even that might be tricky if some IWAD came along that was identified as plain Doom, but actually isn't. (But I'd consider this possibility too unlikely and ultimately irrelevant - making IWADs is not easy and if it really happened a custom IWADINFO entry would be the first thing to be added to the engine.)

Solution 2 seems to be the only viable one, but it's still far from perfect, thanks to the way how most custom content is created.

So, if anyone has better ideas, please speak up!