ZDoom

I think it's probably a missing feature instead of a bug. Okay, will make a new thread about it soon.

[b]Upd:[/b] See [url=https://forum.zdoom.org/viewtopic.php?f=15&t=64268]here[/url]

If there's a problem, please report a bug and provide an example. This uses completely different code for aligning the font and I either need to fix it or map to the generic variant.

Ah, yes, I forgot. The problem was that the characters were incorrectly aligned. I see that there is an enum for that now, but there is no argument for it in the HUDFont constructor.

That already supports monospacing. For the status bar it's a font option.

I'm sorry if I should have started a new thread for this, but since monospacing has been mentioned here before: I see that monospacing support has been added to Screen.DrawText, but what about BaseStatusBar.DrawString?

I'd add such a function if I had sufficient documentation to handle it properly. Ideally, combining diacritics should never reach mod space, unless there is no precomposed alternative. But in the end my knowledge of all this is still far too limited to do it properly. Don't forget that there's also things like variation selectors, that, unlike combining diacritics are not placed AFTER but BEFORE the modified character.

Remember what I said: Unicode processing is a minefield and no matter what you try to cook up yourself will inevitably break if the feature set gets expanded. Although unlikely, what if I added Arabic support? Not only is that a right-to-left script, it also has so many oddities that no left-to-right code trying to process it character by character will ever work. What's there is to analyze a string, not for breaking it apart for printing.

I wonder, doesn't this leave the same problem we had when moving from ascii to utf8? That is, the assumption before was that byte=character, but now multiple bytes can be a character... but the assumption that's likely to be made now is that codepoint=character; what happens if multi-codepoint-characters become a thing in the future (support for combining diacritics is added, say)?
[size=85][edit: I see you've already considered this in the commit message.][/size]

I was thinking, maybe a way to future-proof against this is to also create GetNextCharacter as an alias for GetNextCodePoint. Then, if multi-codepoint-characters are added later, the appropriate changes can be made to GetNextCharacter, while leaving GetNextCodePoint alone. Modders would use whichever one is appropriate: if we want to deal with individual codepoints, we use GetNextCodePoint, and if we want the higher level of abstraction for characters, we use GetNextCharacter, even if they do exactly the same thing for now.

Every function named with "CodePoint" would get a (functionally identical, for now) "Character" alias. These names are also a bit more familiar and probably what people would tend to use who'd want the higher level of abstraction; people who explicitly want to deal with codepoints can use the "CodePoint" versions and be confident that their behavior won't change. What do you think?

I also added some case conversion utilities to the String class which are Unicode-aware, they should be able to handle everything except the Turkish special case for I with dot and i without dot (one place where the Unicode consortium truly messed up by making the case conversion locale dependent, this is nearly impossible to solve unless you know the actual language of the input string - and that it doesn't mix languages.)

All right, thank you again. These new methods will definitely come in handy...

It will return 0 when the string is fully parsed. To convert single characters back to a string, you can use AppendCharacter with an empty source.

Thank you very much. :D

Question: Is there a way to know when GetNextCodePoint has reached the end of the string? Or do I have to use CodePointCount for that?

[b]Edit:[/b] Also, how do I convert the resulting integers to strings to pass them to Screen.DrawText / BaseStatusBar.DrawString etc. ?

Proper handlers have been added for extracting code points from strings.

[quote="Graf Zahl"]The problem with letting this be handled by the community is that it will produce lots of broken code and no means to fix released mods.
Unicode handling is a minefield that seriously cannot be left to amateurs. Regarding monospacing, that really (*REALLY*) needs to be made part of the DrawString function. Even engine-internally, it's currently hacked in in the places that have it, most importantly the status bar's text drawer.[/quote]
Yes, this is exactly why I want the API to be part of the engine. Same about monospacing. Yes, my method is largely a workaround and it indeed might not work correctly with more exotic characters. But right now it doesn't even work with, say, Cyrillic characters. Same problem with capitalization.

Regarding padding: the main problem is that I need to know the lengths of my strings to position them correctly. Whether this is done by forcing them all to the same length with invisible characters or by specifying the correct positions from the start does not really matter. It is also possible that I might need padding with non-invisible characters as well (say, "*" or "-") in certain cases.

The problem with letting this be handled by the community is that it will produce lots of broken code and no means to fix released mods.
Unicode handling is a minefield that seriously cannot be left to amateurs. Regarding monospacing, that really (*REALLY*) needs to be made part of the DrawString function. Even engine-internally, it's currently hacked in in the places that have it, most importantly the status bar's text drawer.

I'm still not really seeing the advantage of doing it engine-side over letting users create a library to support our own needs, letting that mature, and then maybe integrating something similar into the engine once utf-8 support has a little time to settle.

To the extent that the examples like monospacing and uppercasing can be handled by a general purpose utf-8 class engine-side, what would be the benefit of doing it that way over letting the community do it as a separate library? Wouldn't a simple iterator like the example I posted earlier handle a majority of the cases that can actually be handled?