GZStats: A quick rundown

[Rachael]

Thank you! I'll think up of something, soon. We're working on having a more robust server-side component, as well.

[Rachael]

I was expecting personal attacks and hyperbole and a bunch of other bullshit to come with it, and sure enough it happened.

Please do not resort to personal attacks if you expect to stay on this forum long. You can be constructive and offer feedback without calling Graf or the other developers idiots or incompetent, or threatening them.

And remember one thing: GZDoom is being done for free. You did not pay Graf or the other developers a single penny for it. So get off your high horse, you aren't entitled to anything.

[Graf Zahl]

In light of the recent troll I'd just like to make one statement about legacy support, in the hopes that people get an understanding.

No, legacy support is not free of costs. Legacy support requires a lot of work and a lot of testing and a lot of compromising. Even now, the OpenGL 2.0 support takes a heavy toll on the GL 4.x render path. Because OpenGL 2.0 does not support uniform buffers, the entire render state is still being managed with legacy features, even on the most modern hardware. Due to how this works it completely blocks the use of multithreaded processing in the renderer which does have an effect on performance.

And the same is true for each other legacy feature. Nearly all of them have an adverse effect on something relevant for modern systems.

This is the main reason why occasionally some support for old hardware needs to be dropped. If we cannot advance the engine to use the capabilies of modern hardware to the fullest extent possible, eventually the addition of new features will dry up, either because they cannot be fit into the existing framework or because implementing them will become too time consuming. A good example for this is the current existence of 4(!) distinct render paths for doing 2D display, like status bars and HUD messages. And that's simply 3 too many, considering that one is merely a fallback for ultra-ancient hardware and the others could be merged to a large degree, if some minor compromises in supporting some truly ancient hardware were made. Currently they cannot be merged because the needs of some 15 year old hardware that is still on the support list stands in the way.

So what should it be? Keep that support in and forfeit progress or throw it out and make the system usable again? And to make a decision where to draw the line, we need the info of this stats module, so that we do not throw out some support that is still needed.

[drfrag]

Also i've rescued this post since i find it really interesting:

There's one thing that's desperately needed in the engine and that's a unification of the 2D rendering between the HWGL, SWGL and D3D backend. As things are right now, everything that gets added here needs to be done 3 times, PLUS a software fallback, which is an absolutely unacceptable situation for us developers. It virtually means that the 2D code right now is in a total state of limbo because working on it is just too excessive.
However, to streamline it, some sacrifices will have to be made. A long-term viable rewrite here is only doable if the following things get done:

1. Removal of DirectDraw backend.
2. Removal of SM14 support, including the texture atlases in the D3D and SWGL backends
3. Removal of software rendered 2D, except for the fuzzy weapon sprite.
4. Simplification of SWGL to work with lower OpenGL versions (1.4 should be possible, provided that the GL_ARB_texture_non_power_of_two and GL_EXT_shared_palette extensions are available.)

Explanations:
- DirectDraw has been the prime candidate for removal for a long time, but refactoring 2D will not be doable if this stays in.
- Ditching Shader Model 1.4 is an inevitability, because it's just for this ancient hardware that the texture atlases were implemented - but these turn out to be a huge feature blocker for the 2D code because they totally prevent any sane implementation of a flexible system - and aren't worth the effort anyway outside of fonts, but for fonts it makes more sense to create texture atlases on a higher cross-backend level anyway. SM1.4 is a relatively easy feature to remove because only one chip series (ATI R200) ever supported it and support for that ended before XP, so there's no drivers for newer Windows versions.
- software rendered 2D means a separate drawer for everything. But there's basically no hardware out there that really still needs it. If done right SWGL should cover everything remotely relevant on non-Windows and D3D already does on Windows. The only exception may be some rare old cards that cannot do NPOT textures. AFAIK among the still supported hardware that is only Geforce 5xxx series and the higher end of Radeon 9xxx series (i.e. 12+ yeah old hardware).

[Gutawer]

I just wanna note that, to people that are paranoid about this still: Remember that GZDoom is an open-source program. There is no possible way that the devs could sneak in compromising telemetry into GZDoom without people being able to see it. For reference, this is the current implementation of D_DoAnonStats(), the function responsible for forming the GET request which sends user data:

Code: Select all • Expand view

void D_DoAnonStats()
{
   static bool done = false;   // do this only once per session.
   if (done) return;
   done = true;

   // Do not repeat if already sent.
   if (currentrenderer == 0 && sentstats_swr_done >= CHECKVERSION) return;
   if (currentrenderer == 1 && sentstats_hwr_done >= CHECKVERSION) return;

   static char requeststring[1024];
   mysnprintf(requeststring, sizeof requeststring, "GET /stats.php?render=%i&cores=%i&os=%i&renderconfig=%i HTTP/1.1\nHost: %s\nConnection: close\nUser-Agent: %s %s\n\n",
      GetRenderInfo(), GetCoreInfo(), GetOSVersion(), currentrenderer, sys_statshost.GetHumanString(), GAMENAME, VERSIONSTR);
   DPrintf(DMSG_NOTIFY, "Sending %s", requeststring);
   std::thread t1(D_DoHTTPRequest, requeststring);
   t1.detach();
}


The important line here is this:

Code: Select all • Expand view

mysnprintf(requeststring, sizeof requeststring, "GET /stats.php?render=%i&cores=%i&os=%i&renderconfig=%i HTTP/1.1\nHost: %s\nConnection: close\nUser-Agent: %s %s\n\n",
      GetRenderInfo(), GetCoreInfo(), GetOSVersion(), currentrenderer, sys_statshost.GetHumanString(), GAMENAME, VERSIONSTR);


This line inserts a formatted string into the requeststring variable, consisting of a HTTP GET request to the sys_statshost address, with data about rendering info, core info, OS version, and current renderer. It's worth noting that all of these are integers that come from functions, meaning that they're incredibly generic in what they mean. Looking at the definition of GetRenderInfo(), for example:

Code: Select all • Expand view

static int GetRenderInfo()
{
   if (currentrenderer == 0)
   {
      if (!screen->Accel2D) return 0;
      if (vid_glswfb) return 2;
      if (screen->LegacyHardware()) return 6;
      return 1;
   }
   else
   {
      auto info = gl_getInfo();
      if (info.first < 3.3) return 3;   // Legacy OpenGL. Don't care about Intel HD 3000 on Windows being run in 'risky' mode.
      if (!info.second) return 4;
      return 5;
   }
}


This returns an incredibly generic, non-identifying integer telling basic stuff about your rendering capabilities, that will be handled by the GZStats server. It doesn't even send what GPU you're using. I'm not going to do this for every info function, but you can verify for yourself that they are all super-generic. Nothing malicious could be done with any of this data! None of this is even close to the sort of telemetry you see big corporations do on a daily basis to serve you ads and make money. I think it's okay to be skeptical of the idea of data sending, but when we're talking about an open-source piece of software where anything can be checked and verified, it becomes unacceptable to remain ignorant and spread paranoia.

And even if you still don't find this data-sending okay, it's been made opt-in, so I really don't see where the problems here are coming from.
You aren't going to be hurt in any way by this, in fact there is no possible way that you could be, given the nature of the data sent.

[_mental_]

For those who decided / is deciding / will decide to select GZDoom code base for own project, I added CMake configuration option to turn stats sending off.
So, there is even no need to modify source code to completely remove this feature.

[Matt]

May I suggest the following wording for a popup window?

You are using GZDoom [version] for the first time.

You appear to be running GZDoom using the following:

Operating system: __
Processor cores: __
Renderer type: __

May we send this information to gzstats.drdteam.org?

We want to know how much longer we should support this hardware and OS version in the future.

All information sent will be anonymous. We will NOT be sending this information to Microsoft, Facebook, the FBI, your spouse, whoever. It will, however, be sent unencrypted.

If you are getting this notice more than once for this version, or if the above information is incorrect, please let us know on the forums. Thanks!

- the GZDoom dev team


<YES - PLEASE SEND>

<no - I'd rather keep this private>

(as inspired by ZDoom's old crash report button)

(Pressing Enter immediately should not select either option, since people might be pressing Enter without even looking thinking they'd get the IWAD selection window)

[Rachael]

All information sent will be anonymous. We will NOT be sending this information to Microsoft, Facebook, the FBI, your spouse, whoever. It will, however, be sent unencrypted.

Name dropping all of those is a bad idea, in my opinion. There's a point where making a point that we'll not share the information with anyone is more effective than the numerous known information collectors that tend to strike paranoia into the same people we're asking for help from.

But otherwise, that is a very good write-up, in my opinion.

[Matt]

Got it. Just thought it would be nice to have a callback to the old days when the ZDoom crash report (which looked a lot like the Windows one) specifically said in half-jest "We will NOT be sending your information to Microsoft!") - a reminder that what's being done here is not unprecedented in GZDoom's history.

"anyone but the GZDoom devs" then? Or is this going to be publicly available as global stats (like a page with bar graphs somewhere)?

[Graf Zahl]

I don't think that would be a good idea. If people can see the stats some with an agenda may actually try to manipulate them. I think the best course of action is to let it run for a few weeks and then make a post here, once a picture solidifies.

[TheLightBad96]

I only have one question. Has this GZStats been implemented yet and if it has been which builds are using it. I'm referring to the development builds found on the DRD site?

[_mental_]

It’s in since g3.3pre-619-g9b7bef59d.

[Graf Zahl]

It already has given us some really valuable information, the most surprising of which is the low percentage of Windows XP. Being an old game I would have expected this to be a lot higher, but so far it's only less than 1.5%. Right now there's no need to ditch it, but this should serve as a wake-up call to those users still on XP: With such low percentage its days of support are inevitably numbered. The moment an issue arises where XP turns out to be a blocker, it will be gone.

OpenGL 2.0 currently is at roughly 7%, unfortunately still too much to remove support for it. I guess that will have to wait a while longer then. The most likely outcome here is to have a stripped down deprecated renderer that won't see any future advancements as long as there's some need for support.

[Marisa the Magician]

Any news on the native or in-game dialogs?

[_mental_]

Most likely we will use native UI from this branch when user friendly message will be written.