Ability to customize the display of notifications (& more)

Post a reply

Smilies
:D :) :( :o :shock: :? 8-) :lol: :x :P :oops: :cry: :evil: :twisted: :roll: :wink: :geek: :ugeek: :!: :?: :idea: :arrow: :| :mrgreen: :3: :wub: >:( :blergh:
View more smilies

BBCode is OFF
Smilies are ON

Topic review
   

Expand view Topic review: Ability to customize the display of notifications (& more)

Re: Ability to customize the display of notifications (& mor

by _mental_ » Sat Aug 24, 2019 4:21 am

Here is how custom iswalpha() can be implemented. Alternatively, it's possible to code it as a huge number of if's, with or without table creation at runtime.

Re: Ability to customize the display of notifications (& mor

by Graf Zahl » Sat Aug 24, 2019 2:08 am

Considering that even the entire printf family is locale dependent, there's not much that can really be used here. I already had my fun with code breaking because the locale was set to the local country for 'proper' formatting of numbers, but totally overlooked that this renders internal text processing with floats inoperable.

These standards committees are an odd bunch. They seem so fixated at compatibility with theoretical systems that they totally lose focus of practical concerns.
This here's a good point. Apparently they want to keep things compatible with other theoretical encodings by not making an Unicode dependency. Why even use wide chars if not?

Re: Ability to customize the display of notifications (& mor

by dpJudas » Sat Aug 24, 2019 1:18 am

Some of that can be explained by ancient 1980's origins of the C runtime that C++ was based on. The odd thing is that they can't even decide to make unicode part of the standard, because even today you can't actually specify a locale without it being implementation specific.

I completely agree with Chris that the solution is to simply avoid using it at all. For me, std::string is a basic container (that I always put utf-8 into, regardless of the locale) and the only other text related stuff I might use from the standard library only comes into play if its basically locale agnostic (like std::to_string(int)).

Re: Ability to customize the display of notifications (& mor

by Graf Zahl » Fri Aug 23, 2019 3:31 pm

Chris wrote: TL;DR locales in C/C++ suck. Probably better to find a string or character handling library that better understands and works with Unicode/UTF-8/16/32.
Indeed. I wonder why they had to cook up something this totally and fundamentally useless. Most importantly, why is the locale global state? That alone will create all the same problems that have been plaguing OpenGL to the point that it got superseded.

Re: Ability to customize the display of notifications (& mor

by Chris » Fri Aug 23, 2019 3:01 pm

_mental_ wrote:The thing is Unicode standard has a predefined list of alphabetic characters. Have no idea why it’s not the same in all implementations.
Because C++ has a standard locale that's supposed to be set by default on launch (and that locale is defined to clearly specify what is valid for things like is[w]alpha). A better question would be why C or C++ hasn't defined a Unicode locale that any program can set, or update the classic/"C" locale with if it can maintain enough "C" locale compatibility that nothing serious would break.

Though as it is, ctype functions like is[w]alpha are already fundamentally broken for multi-character encoding schemes. isalpha will fail for anything that can't be represented by a single char (extended UTF-8 characters don't work), and iswalpha for anything that can't be represented by a single wchar_t (on Windows, extended UTF-16 characters above 0xffff shouldn't work, and that does include some CJK characters). So unless wchar_t is 32-bit, if a desired codepoint to test is outside of wchar_t's range (e.g. greater than 0xffff on WIndows) it will fail regardless of locale. C++11 added a char32_t type to represent UTF-32 characters without multi-character encoding, but there's no ctype functions that can accept it that don't also require an explicit locale parameter (making it very clunky, though does avoid problems with global state negatively affecting external code).

TL;DR locales in C/C++ suck. Probably better to find a string or character handling library that better understands and works with Unicode/UTF-8/16/32.

Re: Ability to customize the display of notifications (& mor

by Graf Zahl » Fri Aug 23, 2019 1:32 pm

_mental_ wrote: The thing is Unicode standard has a predefined list of alphabetic characters. Have no idea why it’s not the same in all implementations.
My guess is that the C standard goons were overthinking it and the GCC and Clang people dilligently following a broken spec where Microsoft had been doing this 'properly' since 1992, long before there was a coding standard and now can't backpedal because it'd break lots of software.

Re: Ability to customize the display of notifications (& mor

by _mental_ » Fri Aug 23, 2019 1:21 pm

Different library implementations have different set of characters in 'alpha' category. So yes, setting locale isn’t enough as it may fail to detect a letter from some specific character.

The thing is Unicode standard has a predefined list of alphabetic characters. Have no idea why it’s not the same in all implementations.

In such situation, own function is not so bad idea indeed. UnicodeData.txt from the standard contains category information important to us.
Unfortunately, this file has some quirks with a few ranges omitted from it. Tomorrow I plan to make a better fix for this problem.

Re: Ability to customize the display of notifications (& mor

by Graf Zahl » Fri Aug 23, 2019 12:58 pm

Marisa Kirisame wrote:My locale is en_GB.UTF-8, though.
Your locale is not important here, but the fact that the entire GZDoom code was written against the "C" locale. Change that and lots of stuff might break. Software that mucks around with the locale setting has to be extremely careful about the libraries it uses - most simply cannot deal with different locales at all.

Re: Ability to customize the display of notifications (& mor

by Marisa the Magician » Fri Aug 23, 2019 12:00 pm

My locale is en_GB.UTF-8, though.

Also standards are crap IMHO. With the addition of C library and compiler developers (mis)interpreting them in their own special ways too shit's even worse.

Re: Ability to customize the display of notifications (& mor

by Graf Zahl » Fri Aug 23, 2019 4:52 am

_mental_ wrote: The single option I see is to hardcode table for alpha category in order to avoid using iswalpha().

... which makes me wonder, why the hell this isn't even properly defined for a Unicode function. Somehow the C and C++ committees always manage to mess up supposedly simple things. Unicode is quite anal about what category a code point belongs to there, is basically no chance for a letter not being a letter in another language and vice versa.

I guess with such fucked up situations, rolling out one's own table is the only viable solution - like I already did for upper- and lowercase because the language provides no usable facilities.

Re: Ability to customize the display of notifications (& mor

by _mental_ » Fri Aug 23, 2019 4:23 am

Graf Zahl wrote:Are you certain that this doesn't break anything else?
Of course I’m not. But I’m pretty sure that standard conforming iswalpha() cannot be used with C locale either.
Graf Zahl wrote:What precisely does get messed up here?
Like I wrote in the commit message, iswalpha() returns zero for any alphabetic character that isn’t Latin.
That’s why CanPrint returns true for pretty much any argument.
Graf Zahl wrote:I'd rather fix the actual cause than to muck around with C's broken-by-design locale feature.
The single option I see is to hardcode table for alpha category in order to avoid using iswalpha().
Graf Zahl wrote:How does "C.UTF-8" work?
It works on Ubuntu 16.04, although this locale seems to specific for some Linux distros according to Google.

Re: Ability to customize the display of notifications (& mor

by Graf Zahl » Fri Aug 23, 2019 3:29 am

Are you certain that this doesn't break anything else? What precisely does get messed up here? I'd rather fix the actual cause than to muck around with C's broken-by-design locale feature.

How does "C.UTF-8" work?

Re: Ability to customize the display of notifications (& mor

by _mental_ » Fri Aug 23, 2019 3:20 am

Marisa Kirisame wrote:CanPrint doesn't seem to work over here, it apparently always returns true for any string.
It works as expected on Windows though. The problem is caused by differences in locale handling. Selecting UTF-8 locale explicitly fixes this issue.

Re: Ability to customize the display of notifications (& mor

by Marisa the Magician » Tue Aug 20, 2019 5:27 am

Lemme just isolate this.

Edit: here. It should print "new game" in english, russian, japanese and korean, with the included font, and fall back to NewSmallFont if it can't. But all it does is always use the font (which results in nothing drawn for the other 3 lines).

Re: Ability to customize the display of notifications (& mor

by Graf Zahl » Tue Aug 20, 2019 3:05 am

What font, what text?
In the tests I made it correctly recognized if it was printable.

Top