ZDoom

A fully internationalized font would include CJK, and that's not practical.
For supplying a font with support for other languages, here's the checklist spreadsheet: https://docs.google.com/spreadsheets/d/ ... edit#gid=0
Regarding the format, just have a look at the internal fonts, they use a directory under "fonts/" containing all supported glyphs with a 4 digit hex number of their respective code point.
Greek can be ignored for now, the second attempt at translating it fell just as flat as the first one and no real work was done.

Graf Zahl wrote:Unfortunately I didn't have any idea how to handle the internationalization issue. These functions may not print anything themselves, so there's simply no way to do an engine check, I guess the only way is to publicly lambast the makers for being so careless...

The easiest way would probably be to default to the optionsmenu font if a given character is missing from the specified font. This way, the player still gets the appropriate info while OCD-heavy modders get a wake-up call to make their beautiful typeface a little fuller.

Yes, but that needs to be coded into the new output function. Wanna bet that this is the part which most people won't do?
I already added a CanPrint function to the font class.

CanPrint doesn't seem to work over here, it apparently always returns true for any string.

What font, what text?
In the tests I made it correctly recognized if it was printable.

Lemme just isolate this.

Edit: here. It should print "new game" in english, russian, japanese and korean, with the included font, and fall back to NewSmallFont if it can't. But all it does is always use the font (which results in nothing drawn for the other 3 lines).

Marisa Kirisame wrote:CanPrint doesn't seem to work over here, it apparently always returns true for any string.

It works as expected on Windows though. The problem is caused by differences in locale handling. Selecting UTF-8 locale explicitly fixes this issue.

Are you certain that this doesn't break anything else? What precisely does get messed up here? I'd rather fix the actual cause than to muck around with C's broken-by-design locale feature.

How does "C.UTF-8" work?

Graf Zahl wrote:Are you certain that this doesn't break anything else?

Of course I’m not. But I’m pretty sure that standard conforming iswalpha() cannot be used with C locale either.

Graf Zahl wrote:What precisely does get messed up here?

Like I wrote in the commit message, iswalpha() returns zero for any alphabetic character that isn’t Latin.
That’s why CanPrint returns true for pretty much any argument.

Graf Zahl wrote:I'd rather fix the actual cause than to muck around with C's broken-by-design locale feature.

The single option I see is to hardcode table for alpha category in order to avoid using iswalpha().

Graf Zahl wrote:How does "C.UTF-8" work?

It works on Ubuntu 16.04, although this locale seems to specific for some Linux distros according to Google.

_mental_ wrote: The single option I see is to hardcode table for alpha category in order to avoid using iswalpha().

... which makes me wonder, why the hell this isn't even properly defined for a Unicode function. Somehow the C and C++ committees always manage to mess up supposedly simple things. Unicode is quite anal about what category a code point belongs to there, is basically no chance for a letter not being a letter in another language and vice versa.

I guess with such fucked up situations, rolling out one's own table is the only viable solution - like I already did for upper- and lowercase because the language provides no usable facilities.

My locale is en_GB.UTF-8, though.

Also standards are crap IMHO. With the addition of C library and compiler developers (mis)interpreting them in their own special ways too shit's even worse.

Marisa Kirisame wrote:My locale is en_GB.UTF-8, though.

Your locale is not important here, but the fact that the entire GZDoom code was written against the "C" locale. Change that and lots of stuff might break. Software that mucks around with the locale setting has to be extremely careful about the libraries it uses - most simply cannot deal with different locales at all.

Different library implementations have different set of characters in 'alpha' category. So yes, setting locale isn’t enough as it may fail to detect a letter from some specific character.

The thing is Unicode standard has a predefined list of alphabetic characters. Have no idea why it’s not the same in all implementations.

In such situation, own function is not so bad idea indeed. UnicodeData.txt from the standard contains category information important to us.
Unfortunately, this file has some quirks with a few ranges omitted from it. Tomorrow I plan to make a better fix for this problem.

_mental_ wrote: The thing is Unicode standard has a predefined list of alphabetic characters. Have no idea why it’s not the same in all implementations.

My guess is that the C standard goons were overthinking it and the GCC and Clang people dilligently following a broken spec where Microsoft had been doing this 'properly' since 1992, long before there was a coding standard and now can't backpedal because it'd break lots of software.

_mental_ wrote:The thing is Unicode standard has a predefined list of alphabetic characters. Have no idea why it’s not the same in all implementations.

Because C++ has a standard locale that's supposed to be set by default on launch (and that locale is defined to clearly specify what is valid for things like is[w]alpha). A better question would be why C or C++ hasn't defined a Unicode locale that any program can set, or update the classic/"C" locale with if it can maintain enough "C" locale compatibility that nothing serious would break.

Though as it is, ctype functions like is[w]alpha are already fundamentally broken for multi-character encoding schemes. isalpha will fail for anything that can't be represented by a single char (extended UTF-8 characters don't work), and iswalpha for anything that can't be represented by a single wchar_t (on Windows, extended UTF-16 characters above 0xffff shouldn't work, and that does include some CJK characters). So unless wchar_t is 32-bit, if a desired codepoint to test is outside of wchar_t's range (e.g. greater than 0xffff on WIndows) it will fail regardless of locale. C++11 added a char32_t type to represent UTF-32 characters without multi-character encoding, but there's no ctype functions that can accept it that don't also require an explicit locale parameter (making it very clunky, though does avoid problems with global state negatively affecting external code).

TL;DR locales in C/C++ suck. Probably better to find a string or character handling library that better understands and works with Unicode/UTF-8/16/32.