TTS/Screenreader support for blind players

[objectinspace]

I would like to enable the speaking of text from within Doom, so that I can play it better. Specifically, when I am navigating a menu, I should be able to hear the option I am on read back to me. I would also like to be told what item I've just picked up, or the current ammo in a weapon when I switch to it.

Windows has an open-source abstraction library to handle communicating to the screenreader called Tolk: https://github.com/dkager/tolk/
However I have no clue how to integrate this into ZDoom, or how much effort it would take. I'm also not sure what to do about platforms other than windows.

[Graf Zahl]

I'm sorry but this is out of scope for a hobby project.

[Logan MTM]

Did he said "hobby"? Internationalisation was a hobby?

[Rachael]

Internationalisation was a hobby?

Yes. It was purely done by volunteers. And the work they did was amazing.

[Logan MTM]

Well, I have no idea how blind people can play Doom anyway?
Would be really nice if we could set the sound to play when each menu get focus.

[Graf Zahl]

It's not about "blind" people but about "visually impaired people". These are well capable of making out the visuals of the game but cannot read the text content.
But that's not really an easy task in an engine with so many means to output text - some even completely outside the engine's control. Adding such a feature is far from trivial with very little payoff in the end.

[Logan MTM]

I supposed to.
About play specific sounds in Menu?
New game = "sound x"
Load game = "sound y"
Basically the same Pickup.Sound does for items.
Would be great.

[Major Cooke]

Make a new topic for it.

[objectinspace]

Hi, everyone! OP here, need to clarify some things.

It's not about "blind" people but about "visually impaired people". These are well capable of making out the visuals of the game but cannot read the text content.

Though TTS would also be extremely useful for players with low vision (A few hundred million ish people, last I checked) I actually am talking about playing as a completely blind person, because that's what I am. This is why I specifically wrote "blind" in the topic title. Please don't put words in my mouth!

There is actually an accessibility mod for Doom being developed, called the Toby accessibility mod. viewtopic.php?t=71349
It is am impressive effort, but unfortunately it only realizes its full potential for maps specifically designed for it. ZDoom could play an important role here by providing more information for such mods to take advantage of.

But that's not really an easy task in an engine with so many means to output text - some even completely outside the engine's control. Adding such a feature is far from trivial with very little payoff in the end.

???

If empowering the 35 million plus people with sight loss around the world to play Doom has "very little payoff in the end," what is the point of this project? I don't imagine , for example, adding OpenGL/DX11 support to a game that's some 30 years old was trivial either, but it was done, because people wanted it.

I'm actually extremely offended by this answer, on a deep personal level. Your attitude is ignorant, hateful, bigoted, ableist--and most importantly, wrong.

I understand that this does not affect you personally, but that does not mean it isn't worth doing. I also think the implementation will be easier than you think , as there are plenty of other games and emulators--Skullgirls, GTAV, Retroarch, and scumVM come to mind--that have added some form of speech output for blind players, either by the developers or after the fact via modding.

Just my 2 cents!

[Graf Zahl]

???

If empowering the 35 million plus people with sight loss around the world to play Doom has "very little payoff in the end," what is the point of this project? I don't imagine , for example, adding OpenGL/DX11 support to a game that's some 30 years old was trivial either, but it was done, because people wanted it.

How many of these play Doom? You seem to be the first one to come around.
I'm really sorry but this is a hobby project - I simply do not have the time to invest into a feature that may ultimately be used by a handful of users. I surely won't say "no" if someone stepped up to implement it, but as things stand, I see zero chance for it to happen.

[phantombeta]

If empowering the 35 million plus people with sight loss around the world to play Doom has "very little payoff in the end," what is the point of this project? I don't imagine , for example, adding OpenGL/DX11 support to a game that's some 30 years old was trivial either, but it was done, because people wanted it.

GZDoom has many, many ways to display text, a couple of which the engine can't even recognize as being text.
And there's also how doing this naïvely would make older mods (and pretty much any map that wants to draw an image to the screen) constantly spam the letter "A" at you, because the only way to draw images up until pretty recently was by telling it to use a non-font image as a font, then telling it to draw the letter "A". And that's not even going into how none of us (the developers) would really be able to test this properly.

Ultimately, there would indeed be "very little payoff in the end", because most mods wouldn't work properly with it. At best, a couple of mods might work well, but everything else would have issues working with it.
The only way to really have such a thing work acceptably would be to add generic functions for it and have mods implement this themselves, which wouldn't immediately solve the problem, as it'd require modders to make use of it... Which would probably take quite a while to be used in most mods, if ever.

I'm actually extremely offended by this answer, on a deep personal level. Your attitude is ignorant, hateful, bigoted, ableist--and most importantly, wrong.

There's nothing hateful or bigoted about this. No one ever said blind people shouldn't play games, the issue is with implementing it into GZDoom specifically.

I also think the implementation will be easier than you think , as there are plenty of other games and emulators--Skullgirls, GTAV, Retroarch, and scumVM come to mind--that have added some form of speech output for blind players, either by the developers or after the fact via modding.

All of which were certainly developed by groups of people with existing knowledge of how to do this, and most likely without supporting other mods being used on top of the accessibility mods. GZDoom is developed by an extremely small team, none of which receive even donations, and who have zero knowledge of how to do anything like this.

Skullgirls is a commercial game, developed by people who get paid for it, and can even spend money to get it done. GTA V was done by modders who specifically set out to do this, and who most likely already knew what they were doing beforehand. RetroArch and ScummVM are huge projects with dozens of hundreds of contributors. ScummVM even gets assisted with this by having things standardized in the engine.
And unlike GZDoom, these projects aren't a hodgepodge of 20+ years of code (a non-insignificant amount of which was pretty badly designed) that's sometimes only held together by constant maintenance.

If you were willing to find or pay someone else to add and maintain this, sure, we'll try to take it in. But if we have to add and maintain it ourselves, or it causes problems, it simply won't happen, because it isn't feasible for us to do so. Please don't think we don't want this; there's simply technical difficulties with supporting such a thing in this specific engine.

[Edit]: Added the OP's username to the quote headers.

[Rachael]

I'm going to add to what Graf said here, speaking only for myself: (and this was all typed before phantombeta posted her response, so this post is ignorant of anything she said, sorry)

If it's as simple as hooking up an API to a ZScript function so that a modder can take over from there, then I can do that and am willing to do it.

But let's get down to brass tacks here: I am not visually impaired, and neither is anyone I live with. Yes, I take my sight for granted. I have no idea what I would do without it. And with that in mind, I am not unsympathetic to people who have vision loss, it's just that I have no idea what they need, and quite frankly it would be more insulting for me to try and guess at it than it would be to let someone make a mod who's much more capable and knowledgeable than me to offer such assistance to said folks, to do so in such a mod.

So that would be my answer. In the end it is a hobby project, and no I am not going to spend a week researching this topic (especially because I am fairly sure I'll get it wrong anyway if I do - I am just too lacking in knowledge on this subject, that's all there is to it).

[peewee_RotA]

The menus are all read from Language files right? Wouldn't this just require sending that language token off to some other method when certain events happen? (i.e. click or arrow key)

I see that the keys are handled through this CurrentMenu->CallResponder method. That CurrentMenu object has got to have a reference to the i18n token. I'm sure with just a little bit more familiarity, it would be a good item for a community member to look into.

Totally understandable to not try to take it on as a main development feature, but for a curious programmer from the community, this would be a great task.

[SanyaWaffles]

I'm just going to say, this is probably a post that is going to ruffle some feathers. Mainly because I have plenty of experience with this.

First, as Phantombeta said, it's a matter of there's multiple ways text is rendered and processed. We have to account for mods, and since GZDoom is a game that supports a plethora of modded content - we have to make backwards compatibility a paramount effort. This would singlehandedly, with the scope of the project be the biggest undertaking of the engine - bigger than probably getting it GPLv3 compliant.

One of my partners is blind and she contributes to the work we do extensively at our studio. I haven't asked her about this specific topic (GZDoom hooking into screen readers), but she seems to enjoy just having a more auditory experience when we stream the game our project makes extensive use out of voice acting, rather than relying on solely text-related prompts like most games do.

I know it devastated her to lose her sight and put her into a deep depression, which sometimes hits her like a ton of bricks, and she wishes she could engage in gaming again. This is why I do my best to work within the GZDoom engine to make it so people like Jenna don't have to rely on just a reading experience. And she's completely blind - only able to see basic colors and light. She can't make out shapes. She's been like this since 2017. It's one of the reasons SHDX had such extensive use of voice acting, and we're continuing that for Project Absentia.

My opinion on the matter - mainly from talking to Jenna extensively about her being blind and what she needs from a user friendly experience - is this is a problem with a ton of open source projects - there's not enough funding and not enough dedicated people, and people tend to get it wrong even in an open source environment. And it can be even worse in a corporate environment. You think it's bad in the open source community? Look at Twitch - a company owned by a gigantic monolithic corporation that bleeds their workers dry. Their idea of accessibility? Fucking censoring people because "blind playthrough" is somehow ableist. She doesn't agree with that ruling - and I'm going to listen to Jenna more than some random Twitterite on the matter, frankly. And you got corporations rather taking the issue all the way to the supreme court than funneling money to make their goddamn app accessible to NVDA or Talkback or whatever you use.

That said, she also knows small hobby engines like GZDoom don't have the manpower to implement something like this - it's a community effort, but none of us, myself included, are experienced with how to hook something almost entirely graphical (with no windows form UI to speak of) into something like a screen reader. The best way to do it would be to either fork the engine yourself with people who know what needs to be implemented, or do as I'm doing - work within the framework of the engine and use stuff like ZScript (which powers most of the menus and most game logic) to make the user experience less reliant on text alone.

The thing is, it can be done if people were willing to put effort into it. People who know what they're doing. People who'll listen to people with poor sight - and do more than just polish up mere language. However, it's something that's hard to do when you have a ton of dedicated time and there's all sorts of ways it could be impemented - and it could end up wrong.

I do want to add something though. And this might come across harsh.

I'm actually extremely offended by this answer, on a deep personal level. Your attitude is ignorant, hateful, bigoted, ableist--and most importantly, wrong.

I know I'm not the devs thus it was not directed at me, but assuming everyone here is all these "isms" is fucking stupid, frankly - and I don't care if the devs or the mod teams get angry at me, I'm not going to stand for this.

There are people here who know this shit - I deal with Jenna's condition daily, and she's been having issues with her hearing on top of all this - and I'm deeply worried I'm going to lose her. It breaks my heart. She's never seen my face, and I've visited her twice! And it's a fucking pain to work with apps that aren't coded properly - so it's a crapshoot if NVDA will even work with whatever we have to use for collaboration. I was on a podcast with Jenna recently, and it was 50/50 that NVDA would work with the website we used to do the podcast.

And yes, in WIS, I do tailor the apps we use to accommodate her lack of sight. Even then it's not 100%. It's making me seriously considering moving to Office once we get enough funding just so she can have accessibility - Google Docs ain't cutting it as much as I'd like it to.

So I get it - I fucking get it so goddamn much. It makes me really goddamn sad to see her go through this shit and the only thing I can do is, within my knowhow and what she tells me, try to make her contributions at Waffle Iron Studios count, and to do my best to make my projects accessible to her so she can "see" the fruits of her labor rather than never be able to experience what we've made.

I don't consider it a hindrance. She's my best friend and I love her dearly. And I'm doing the best I can within my means to make things accessible for her. However, it pains me there's only so much I can fucking do with a limited budget. If I knew someone and had billions to spare, you bet your britches I'd point them towards GZDoom and go "hey, make this hook into NVDA please"

I'm getting quite tired of people just assuming because I can't just snap my fingers and make shit happen personally dragging out Jeff Bezos' to sieze his means of production myself, or making everything accessible when I have no goddamn money yet and my programming skills aren't the best aside working with a fork of an engine older than most of my friends, I'm somehow a bigoted monster who wants minorities to fry or some shit. I have fucking limits. I'm just a single person. And I'm tired of this goddamn mentality being prevalent everywhere I go. People assuming the worst out of everyone. It frankly needs to stop.

EDIT: Talking to Jenna about this just now. She says the best way would be to hook a TTS into GZDoom itself, but that'd be such massive overhead from what she's describing to me, and still would be a monumental task. Not to mention, it'd make the software massive unless you choose a proper library for it, or use dedicated speech APIs per system. However, that might be horrible to code given we target tons of OSes - and they all probably have their own.

I also want to state, there is UI scaling available, and I'm working to make sure my project works with this properly.

[MartinHowe]

I think the problem here is the issue of bolting on a modern feature to a game architecture written before even the 1995 Disability Discrimination Act, let alone the 2010 Equality Act (using UK legislation for illustration here). Back in 1993, awareness among the general population - presumably including Id's programmers in their capacities as such - was low. As time has progressed, our understanding as a society has improved a bit; we now have laws to help disabled people and computer programming standards are more aligned to the needs of diverse people. The problem is, I feel, two-fold.

Firstly, the architecture of the Doom Engine is fundamentally 8-bit ASCII with images used to contain text (e.g., M_JKILL). Heretic improved that with the ability to write these as text from a doom-laden font (pardon the pun ) Graf Zahl and others have added things like Unicode support. The engine has, generally speaking, evolved over time. However, to do EQD properly requires software to be designed from the ground up with it in mind, with all the hooks into the engine to support it. In the case of software with user mods, modders need to support it as well. Since Doom is 28 years old, it's simply too late. There are literally thousands of mods out there that were written by kids who had never even heard of EQD, at at time when, depending on their location and first language, the term EQD didn't even exist.

Secondly, the vast amount of effort and formality needed to support EQD properly is way beyond the resources of the average hobbyist or small team. Big megacorps and larger regular companies can afford teams of people who do nothing but oversee this sort of thing, compliance officers who check software and procedures to ensue they are EQD friendly, etc. Small companies and hobbyists simply don't have the time to do this as there are so many different disabilities, health conditions, races, languages, etc., to consider.

So this is really a resource issue, plus a historical legacy issue. As a disabled person myself, I would have had a much better life if I'd been born in 1995 instead of 1965, but there's nothing I can do about it; only hope (and, in my job, work) for a better life for people like me in 50 year's time. It's harsh, but that's the reality of it. I think better support for visually, or other, impaired people is possible to do in an old game, but it will never be as complete as a game designed with today's sensibilities in mind, and more importantly, it needs enough volunteers with time on their hands to do it.