[Fixed] [4.1.3] Controller options appear in chinese

Bugs that have been investigated and resolved somehow.

Moderator: GZDoom Developers

[4.1.3] Controller options appear in chinese

Postby drfrag » Sat Jun 29, 2019 6:13 am

In old versions the name of the controller (logitech) and the axes were in spanish, language selection (auto or enu) doesn't matter.
Now they are in chinese? with the exception of the X, Y and Z letters. There are some square characters with numbers inside like matrices too.
In the vintage build there are no specific names i guess becouse i didn't pick those characters.
I guess may be it's due to the removal of the automatic language detection name not being in unicode format?
Last edited by drfrag on Sat Jun 29, 2019 7:57 pm, edited 1 time in total.
User avatar
drfrag
I.R developer, I.R smart
Vintage GZDoom Developer
 
Joined: 23 Apr 2004
Location: Spain

Re: [4.1.3] Controller options appear in chinese

Postby drfrag » Sat Jun 29, 2019 1:12 pm

chinese.png

Language selection doesn't matter.
User avatar
drfrag
I.R developer, I.R smart
Vintage GZDoom Developer
 
Joined: 23 Apr 2004
Location: Spain

Re: [4.1.3] Controller options appear in chinese

Postby Rachael » Sun Jun 30, 2019 3:40 am

drfrag wrote: I guess may be it's due to the removal of the automatic language detection name not being in unicode format?

Yeah, I was gonna say... what makes you think the automatic language detection has anything to do with it? But then I saw you strike that out.

I had a peek at the i_dijoy.cpp file, and needless to say it's a very ancient work that has not been updated in a very long time.

If it is what you stated you suspect it is, however, it would not be consistently kanji script. Kanji takes up a very small portion of the Unicode table, and if there was an issue converting from CP 1272 to Unicode then the letters would be seemingly random from all over the place.

The automatic language detection removal seems to be a more likely culprit, but I have no way to confirm this right now. The joystick code is calling WMI to get the list of joysticks. It still has ancient code in there to support Windows 2000 that should probably be removed at this point.

I suspect this is one of those things that's not going to be fixed unless it's by a Windows developer who has a controller.
User avatar
Rachael
Webmaster
 
Joined: 13 Jan 2004
Discord: Rachael#3767
Twitch ID: madamerachelle
Github ID: madame-rachelle

Re: [4.1.3] Controller options appear in chinese

Postby phantombeta » Sun Jun 30, 2019 4:59 am

It's actually some mix up between non-UTF-16 and UTF-16. Windows uses UTF-16 for a bunch of stuff, so it's the most likely culprit.
Taking the correct text, saving it into a text file as UTF-8, then opening the file as UTF-16 LE in Sublime Text gives some very similar results to what we see in GZDoom:

Image
Not quite the exact same, but it's invalid input, anyway, so differences are to be expected.
User avatar
phantombeta
In the meadow of sinful thoughts, every flower's a perfect one
 
Joined: 02 May 2013
Location: The United Soviet Socialist Dictatorship of Hueland
Discord: phantombeta#2461
Twitch ID: phantombeta_
Github ID: Doom2fan
Operating System: Windows 10/8.1/8 64-bit
Graphics Processor: nVidia with Vulkan support

Re: [4.1.3] Controller options appear in chinese

Postby dpJudas » Sun Jun 30, 2019 5:43 am

As far as I know, Graf changed the internal representation in GZDoom strings to be UTF-8. IID_IDirectInput8A does not return UTF-8 strings, it returns ANSI strings. The code in i_dijoy.cpp simply assigns the values as-is to its FString.

Based on the stuff phantombeta wrote, it looks like there are two missing character conversions? First from ANSI to UTF-8, then from UTF-8 to UTF-16?

The code in i_dijoy.cpp seems a bit crazy in general - mixing dinput 3, dinput 8, xinput, WMI and rawinput. What could possibly go wrong? ;)
dpJudas
 
 
 
Joined: 28 May 2016

Re: [4.1.3] Controller options appear in chinese

Postby drfrag » Sun Jun 30, 2019 1:52 pm

After looking a bit into it i noticed it could not be the language autodetection thing. Besides i don't know chinese or japanese? but i don't think those people write using matrices. :) So it didn't look like real chinese.
I know we need another version of the interface or doing a conversion or two, is there a function to do the conversion in the engine? As expected i looked at the code but i didn't get very far.
Most likely i've got no idea of what i'm talking about but what about using IID_IDirectInput8W instead?
User avatar
drfrag
I.R developer, I.R smart
Vintage GZDoom Developer
 
Joined: 23 Apr 2004
Location: Spain

Re: [4.1.3] Controller options appear in chinese

Postby dpJudas » Sun Jun 30, 2019 2:15 pm

OK, I plugged in my Logitech joystick, set up some breakpoints and pushed a fix.

The issue was that UNICODE is defined, which in turn defines IDirectInput8 to IDirectInput8W instead of IDirectInput8A. The original code clearly assumed this wasn't the case. It used an explicit query for the ANSI interface, which was no longer correct.
dpJudas
 
 
 
Joined: 28 May 2016

Re: [4.1.3] Controller options appear in chinese

Postby drfrag » Sun Jun 30, 2019 2:32 pm

Well done. Good to see i was not very far off the mark this time. :)
User avatar
drfrag
I.R developer, I.R smart
Vintage GZDoom Developer
 
Joined: 23 Apr 2004
Location: Spain

Re: [4.1.3] Controller options appear in chinese

Postby Graf Zahl » Mon Jul 01, 2019 1:28 am

dpJudas wrote:OK, I plugged in my Logitech joystick, set up some breakpoints and pushed a fix.

The issue was that UNICODE is defined, which in turn defines IDirectInput8 to IDirectInput8W instead of IDirectInput8A. The original code clearly assumed this wasn't the case. It used an explicit query for the ANSI interface, which was no longer correct.



Up until the localization effort, all low level ZDoom code was strictly ANSI. Thanks to lots of type casting and generally bad style of the Windows API it's not really surprising that something was overlooked.
The #ifdef UNICODE is quite redundant, though, it cannot be undefined anymore. I don't really understand why nearly all the Windows tutorials stlll try to convey the dual-format approach and just don't simply say "Unicode is the only way"... :?
User avatar
Graf Zahl
Lead GZDoom Developer
Lead GZDoom Developer
 
Joined: 19 Jul 2003
Location: Germany

Re: [4.1.3] Controller options appear in chinese

Postby dpJudas » Mon Jul 01, 2019 3:46 am

In my own programs I don't support ANSI (targeting both only made sense around Windows 95 where the unicode support was a redistributable and memory extremely scarce). In this case I only added the #ifdef because the code was written in such an old form to begin with.

As for tutorials, I think it is ultimately because most people don't understand character sets very well. Or why the dual form existed in the first place (for porting from DOS). If they did, they would use the modern approach which is to decide either to use internal strings as UTF-8 (std::string) or UTF-16 (std::wstring) and then add two functions for converting to and from that into the other form. ANSI is never the correct solution.
dpJudas
 
 
 
Joined: 28 May 2016

Re: [4.1.3] Controller options appear in chinese

Postby Graf Zahl » Mon Jul 01, 2019 4:37 am

I wouldn't use std::wstring because it's not portable - on most POSIX systems wchar_t is a 32 bit type.
That said, basically the only software still using ANSI is either crappy enterprise software that has lived long enough or backwards looking Open Source, like old games where the developers often suffer from some irrational desire to continue supporting Windows 9x at the expense of modernization.

GZDoom only had it this long because there was no pressing need to change.
User avatar
Graf Zahl
Lead GZDoom Developer
Lead GZDoom Developer
 
Joined: 19 Jul 2003
Location: Germany

Re: [4.1.3] Controller options appear in chinese

Postby dpJudas » Mon Jul 01, 2019 4:55 am

Good point about the wchar_t for the portability - didn't know that. I only used std::wstring in one project, which was strictly Windows where it worked fine.

Nowadays I always use UTF-8 for everything as I believe that was the encoding that ultimately won out as the superior way. The null bytes and endianess issues of UTF-16 makes it unattractive. I think if Win32 had been designed today they would have gone with UTF-8 as well, but at the time it seemed like a good idea as UCS-2 was the original size of the unicode character set.
dpJudas
 
 
 
Joined: 28 May 2016

Re: [4.1.3] Controller options appear in chinese

Postby Graf Zahl » Mon Jul 01, 2019 5:32 am

dpJudas wrote:Nowadays I always use UTF-8 for everything as I believe that was the encoding that ultimately won out as the superior way. The null bytes and endianess issues of UTF-16 makes it unattractive.


Me, too. I made one exception in GZDoom where I use u32string for the console's command line - in this case it was more convenient to have equal code point size than saving space.


dpJudas wrote: I think if Win32 had been designed today they would have gone with UTF-8 as well, but at the time it seemed like a good idea as UCS-2 was the original size of the unicode character set.


Most likely, yes. This was a classic case of an early adopter being screwed by how things developed.
What Microsoft can be faulted for is not adjusting for far too long. Now it's going to be a long time until the last ANSI traces get eliminated from existing software.
User avatar
Graf Zahl
Lead GZDoom Developer
Lead GZDoom Developer
 
Joined: 19 Jul 2003
Location: Germany


Return to Closed Bugs

Who is online

Users browsing this forum: No registered users and 2 guests