The importance of simple examples in bug reports

Here, developers communicate stuff that does not go onto the main News section or the front page of the site.
[Dev Blog] [Development Builds] [Git Change Log] [GZDoom Github Repo]

Moderator: GZDoom Developers

Post Reply
User avatar
Rachael
Posts: 13531
Joined: Tue Jan 13, 2004 1:31 pm
Preferred Pronouns: She/Her
Contact:

The importance of simple examples in bug reports

Post by Rachael »

Today I want to touch on a topic that I don't think is stressed enough in this community: And it applies, in reality, to every project, not just Doom-related ones, and certainly not just GZDoom and its relatives regarding bug reports: Examples.

A lot of times, someone will post a bug report, which doesn't have a lot of info in it, and is pretty much the equivalent of "hurr hurr it duzint wurk!" These bugs are extremely difficult to solve, and the chances of it even being looked at are far less when you make it more difficult for it to be investigated.

While it is our goal to solve every bug we can, not leaving us with enough information about it is hugely problematic.

Another issue that seems to happen a lot is very complex mods are given as examples, requiring a huge download (and sometimes even a click-maze of advertisements to get through). This not only increases the amount of time required for us to acquire said mod in order to investigate it, but it greatly increases the difficulty of fixing the issue because of the amount of effort required to actually play the mod to get the issue to occur.

Two things really get in the way of fixing a mod: Startup time (with bigger mods) and time taken to reproduce the bug. The reason why this is such a huge issue is because we're not wholly unlike modders, ourselves - except our work requires us to restart GZDoom repeatedly because there is no way to make "live" changes to the running game. To make matters worse, sometimes we have to use debug builds, which are less than half the speed of the builds everyone else uses (that we distribute, both as releases and as devbuilds) - so complex mods are completely out of the question because it literally *is* unplayable for us, not to mention a nightmare to load. Repeatedly.

Here is an example of a GOOD bug report:
viewtopic.php?f=7&t=51976

The example is simple. It is nothing more than a square room that plainly shows the problem. There is no jumping through hoops to get it, it's simple to acquire, simple to run, and the problem shows immediately upon entering the map. This is ideal - and it should take you no more than a few minutes to construct. It doesn't have to be pretty - feel free to use the default textures. We won't judge your mapping skills on it. We'd really like to see more reports like these.

Here is an example of a BAD bug report:
1. Load up WolfenDoom: Blade of Agony with a recent GZDoom build.
2. Go to INTERMAP. Optionally, play through some of the missions until Marlene Dietrich appears.
3. Save your game.
4. Re-load your saved game.
5. The time of day and/or weather may change, or Marlene Dietrich and her audience will vanish.
There are a number of problems with this: First off, this is a huge mod, requiring a significant download in order to investigate. Secondly, it requires us to actually play missions - with an unknown length of time, which is *NOT* good when we have to restart the game over and over again, especially in a Debug build! The worst part is, it's difficult to know just by something random like a change in weather whether we solved the bug or not. I really do not want to see examples like these because they really hinder our ability to investigate them. (The original author was kind enough to whip up a more simplified example - thanks for that, and sorry to use your report as an example, but it is just one of many that are problematic)

Another problem with this is the instructions state "recent GZDoom build." If we aren't able to solve the bug immediately, then what happens is "recent" becomes highly ambiguous - to the point where we discard the bug report because we don't know what version to use. (In fact, it may have already been solved by then) Look in your GZDoom-username.ini file on the top line - it has the following:

Code: Select all

# This file was generated by GZDoom g2.4pre-693-g4a87a59 on Sat Mar 11 02:18:13 2017
Use that line to get the game's version number. If you don't know what part we want, just paste the whole line. It's okay! We know what to look for, there.

What it really comes down to is - we're not clairvoyant, and we need YOUR help in order to be able to help YOU! Keep it simple, make it so that we can restart the game over and over again right into the problem until it is solved. That will help HUGELY when we try and fix the bug.

If you do need to provide a big mod, however (and the problem is not feasible in a smaller one), giving console commands (such as player coordinates) or a savegame that warps right to the issue is helpful, also.

Thank you!
User avatar
Major Cooke
Posts: 8170
Joined: Sun Jan 28, 2007 3:55 pm
Preferred Pronouns: He/Him
Location: QZDoom Maintenance Team

Re: The importance of simple examples in bug reports

Post by Major Cooke »

Rachael wrote:If you do need to provide a big mod, however (and the problem is not feasible in a smaller one), giving console commands (such as player coordinates) or a savegame that warps right to the issue is helpful, also.
I'm glad this was touched upon because sometimes I encounter a situation where I've made a perfect replica, and the damn thing won't occur for some stupid reason.

I've ripped hair out of my scalp for (slightly) less.
User avatar
Rachael
Posts: 13531
Joined: Tue Jan 13, 2004 1:31 pm
Preferred Pronouns: She/Her
Contact:

Re: The importance of simple examples in bug reports

Post by Rachael »

Even in that case, if you can remove anything in the map to make the issue more accessible, it helps a lot. :)
User avatar
Zhs2
Posts: 1269
Joined: Fri Nov 07, 2008 3:29 pm
Graphics Processor: ATI/AMD with Vulkan/Metal Support
Location: Maryland, USA, but probably also in someone's mod somewhere
Contact:

Re: The importance of simple examples in bug reports

Post by Zhs2 »

Should also be mentioned that providing a savegame is extremely helpful if the issue is related to a certain length of playtime or the save parser itself.

In general, anything that cuts down on time taken to reproduce is helpful - I feel like that should be the baseline of your wall of text up there. I can recall a number of issues I thought were bugs until stripping out everything unrelated revealed that I misunderstood a function and actually committed user error, which actually helped me to understand things about the engine's little intricacies I would never have known before. I would like to think there are more steps to advocate than "Keep It Simple, Stupid!" (even though this is a very good rule in and of itself) so that people better understand the submission process. It is a process, after all - figure out what went wrong and where, gather applicable materials, make a test case, leave it alone and come back with a fresh mind if you can't, submit everything relevant in a clear and concise bug report if you can, etc. Even simple as a rule can't possibly cut every bug report, and simple doesn't stop people from shooting off a report without verifying it (or, as a direct consequence, failing to explain how their short yet baffling bug report can be considered simple).
User avatar
Akira_98
Posts: 213
Joined: Sun Jun 27, 2010 2:45 pm
Location: AC District 7

Re: The importance of simple examples in bug reports

Post by Akira_98 »

A bit late to this, but oh well.

At what point do you just not want a bug report because it's too vague? Having come across a bug or two that I just outright could not reproduce reliably, there's definitely a point where the average user is not going to be able to provide you with exactly what's happening. I have maybe a passing recognition of DECORATE circa custom damage types, that's not really enough to boil down a freeze in a mod like DoomRL Arsenal to the exact cause if it's not something immediately obvious. That's going to get significantly worse with ZScript, which can't just be understood by referencing a handful of wiki pages.
User avatar
Graf Zahl
Lead GZDoom+Raze Developer
Lead GZDoom+Raze Developer
Posts: 49056
Joined: Sat Jul 19, 2003 10:19 am
Location: Germany

Re: The importance of simple examples in bug reports

Post by Graf Zahl »

Here's a good example of a report that is as good as worthless:

https://mantis.zdoom.org/view.php?id=420

Such a thing simply cannot be investigated.
The other issue is with large mods. It is often not discernible if a problem comes from poor mod coding or a genuine engine bug. And these are a nightmare to test. That's where cut down example mods come in. They not only make it easier to focus on the actual problem, but also rule out that the mod plays foul.
User avatar
Xaser
 
 
Posts: 10772
Joined: Sun Jul 20, 2003 12:15 pm
Contact:

Re: The importance of simple examples in bug reports

Post by Xaser »

I think this announcement focuses a bit too much on the case where the person who's filing the bug report is also a modder. I'd wager that's a pretty big percentage, but it lacks directions for the folks who wouldn't know how to put together an example wad.

For the lay-players out there, perhaps we should list the following bits of advice:
  • Exact version numbers are pretty much mandatory.
  • If the issue is a crash or something graphics related, OS and hardware should be listed.
  • List exact steps to reproduce. Things like "Optionally, play some of the missions" aren't any good. :P
  • Savegames are handy.
  • If all else fails, consider notifying the person who made the mod, as they may be better equipped to debug it.
    • Same advice applies to the devs -- if debugging a mod issue is going to be troublesome, roping in the modder may ease the burden.
If it's possible to cut out complex mods from the equation, that's best, but it's just not always possible. I just don't want to start seeing reports closed because "mod 'X' is too big" and no other reason -- "On Hold", sure though. ;)
User avatar
AFADoomer
Posts: 1322
Joined: Tue Jul 15, 2003 4:18 pm
Contact:

Re: The importance of simple examples in bug reports

Post by AFADoomer »

Would it be possible to print the engine version to the console on startup? That way dumping a log file would guarantee getting the version number...
User avatar
Rachael
Posts: 13531
Joined: Tue Jan 13, 2004 1:31 pm
Preferred Pronouns: She/Her
Contact:

Re: The importance of simple examples in bug reports

Post by Rachael »

I am bumping this because this is becoming an issue again.

D4D and WolfenDoom:BOA are NOT suitable test examples in bug reports! Please do not link these mods.

Please only link these when there's a very specific issue that can absolutely, no matter how many times you attempt it, be reproduced with a stripped-down minimal example!

If your issue pertains to monsters or decorations and you don't want to include a map, simply replace the zombieman actor. We'll load them in with E1M1 or MAP01 - no issue there.
Post Reply

Return to “Developer Blog”