The importance of simple examples in bug reports

Here, developers communicate stuff that does not go onto the main News section or the front page of the site.

Moderator: Developers

The importance of simple examples in bug reports

Postby Rachael » Sat Mar 11, 2017 4:59 pm

Today I want to touch on a topic that I don't think is stressed enough in this community: And it applies, in reality, to every project, not just Doom-related ones, and certainly not just GZDoom and its relatives regarding bug reports: Examples.

A lot of times, someone will post a bug report, which doesn't have a lot of info in it, and is pretty much the equivalent of "hurr hurr it duzint wurk!" These bugs are extremely difficult to solve, and the chances of it even being looked at are far less when you make it more difficult for it to be investigated.

While it is our goal to solve every bug we can, not leaving us with enough information about it is hugely problematic.

Another issue that seems to happen a lot is very complex mods are given as examples, requiring a huge download (and sometimes even a click-maze of advertisements to get through). This not only increases the amount of time required for us to acquire said mod in order to investigate it, but it greatly increases the difficulty of fixing the issue because of the amount of effort required to actually play the mod to get the issue to occur.

Two things really get in the way of fixing a mod: Startup time (with bigger mods) and time taken to reproduce the bug. The reason why this is such a huge issue is because we're not wholly unlike modders, ourselves - except our work requires us to restart GZDoom repeatedly because there is no way to make "live" changes to the running game. To make matters worse, sometimes we have to use debug builds, which are less than half the speed of the builds everyone else uses (that we distribute, both as releases and as devbuilds) - so complex mods are completely out of the question because it literally *is* unplayable for us, not to mention a nightmare to load. Repeatedly.

Here is an example of a GOOD bug report:
viewtopic.php?f=7&t=51976

The example is simple. It is nothing more than a square room that plainly shows the problem. There is no jumping through hoops to get it, it's simple to acquire, simple to run, and the problem shows immediately upon entering the map. This is ideal - and it should take you no more than a few minutes to construct. It doesn't have to be pretty - feel free to use the default textures. We won't judge your mapping skills on it. We'd really like to see more reports like these.

Here is an example of a BAD bug report:
1. Load up WolfenDoom: Blade of Agony with a recent GZDoom build.
2. Go to INTERMAP. Optionally, play through some of the missions until Marlene Dietrich appears.
3. Save your game.
4. Re-load your saved game.
5. The time of day and/or weather may change, or Marlene Dietrich and her audience will vanish.


There are a number of problems with this: First off, this is a huge mod, requiring a significant download in order to investigate. Secondly, it requires us to actually play missions - with an unknown length of time, which is *NOT* good when we have to restart the game over and over again, especially in a Debug build! The worst part is, it's difficult to know just by something random like a change in weather whether we solved the bug or not. I really do not want to see examples like these because they really hinder our ability to investigate them. (The original author was kind enough to whip up a more simplified example - thanks for that, and sorry to use your report as an example, but it is just one of many that are problematic)

Another problem with this is the instructions state "recent GZDoom build." If we aren't able to solve the bug immediately, then what happens is "recent" becomes highly ambiguous - to the point where we discard the bug report because we don't know what version to use. (In fact, it may have already been solved by then) Look in your GZDoom-username.ini file on the top line - it has the following:

Code: Select allExpand view
# This file was generated by GZDoom g2.4pre-693-g4a87a59 on Sat Mar 11 02:18:13 2017


Use that line to get the game's version number. If you don't know what part we want, just paste the whole line. It's okay! We know what to look for, there.

What it really comes down to is - we're not clairvoyant, and we need YOUR help in order to be able to help YOU! Keep it simple, make it so that we can restart the game over and over again right into the problem until it is solved. That will help HUGELY when we try and fix the bug.

If you do need to provide a big mod, however (and the problem is not feasible in a smaller one), giving console commands (such as player coordinates) or a savegame that warps right to the issue is helpful, also.

Thank you!
User avatar
Rachael
 
Joined: 13 Jan 2004

Re: The importance of simple examples in bug reports

Postby Major Cooke » Sat Mar 11, 2017 5:17 pm

Rachael wrote:If you do need to provide a big mod, however (and the problem is not feasible in a smaller one), giving console commands (such as player coordinates) or a savegame that warps right to the issue is helpful, also.


I'm glad this was touched upon because sometimes I encounter a situation where I've made a perfect replica, and the damn thing won't occur for some stupid reason.

I've ripped hair out of my scalp for (slightly) less.
User avatar
Major Cooke
"Shit happens, but I'd love it if it didn't." -AEoD Team
 
Joined: 28 Jan 2007

Re: The importance of simple examples in bug reports

Postby Rachael » Sat Mar 11, 2017 5:22 pm

Even in that case, if you can remove anything in the map to make the issue more accessible, it helps a lot. :)
User avatar
Rachael
 
Joined: 13 Jan 2004

Re: The importance of simple examples in bug reports

Postby Zhs2 » Sat Mar 11, 2017 11:51 pm

Should also be mentioned that providing a savegame is extremely helpful if the issue is related to a certain length of playtime or the save parser itself.

In general, anything that cuts down on time taken to reproduce is helpful - I feel like that should be the baseline of your wall of text up there. I can recall a number of issues I thought were bugs until stripping out everything unrelated revealed that I misunderstood a function and actually committed user error, which actually helped me to understand things about the engine's little intricacies I would never have known before. I would like to think there are more steps to advocate than "Keep It Simple, Stupid!" (even though this is a very good rule in and of itself) so that people better understand the submission process. It is a process, after all - figure out what went wrong and where, gather applicable materials, make a test case, leave it alone and come back with a fresh mind if you can't, submit everything relevant in a clear and concise bug report if you can, etc. Even simple as a rule can't possibly cut every bug report, and simple doesn't stop people from shooting off a report without verifying it (or, as a direct consequence, failing to explain how their short yet baffling bug report can be considered simple).
User avatar
Zhs2
Power of three.
 
Joined: 07 Nov 2008
Location: Maryland, USA

Re: The importance of simple examples in bug reports

Postby Akira_98 » Wed Mar 15, 2017 5:20 am

A bit late to this, but oh well.

At what point do you just not want a bug report because it's too vague? Having come across a bug or two that I just outright could not reproduce reliably, there's definitely a point where the average user is not going to be able to provide you with exactly what's happening. I have maybe a passing recognition of DECORATE circa custom damage types, that's not really enough to boil down a freeze in a mod like DoomRL Arsenal to the exact cause if it's not something immediately obvious. That's going to get significantly worse with ZScript, which can't just be understood by referencing a handful of wiki pages.
User avatar
Akira_98
Well excuse me, princess.
 
Joined: 27 Jun 2010
Location: AC District 7

Re: The importance of simple examples in bug reports

Postby Graf Zahl » Wed Mar 15, 2017 5:55 am

Here's a good example of a report that is as good as worthless:

https://mantis.zdoom.org/view.php?id=420

Such a thing simply cannot be investigated.
The other issue is with large mods. It is often not discernible if a problem comes from poor mod coding or a genuine engine bug. And these are a nightmare to test. That's where cut down example mods come in. They not only make it easier to focus on the actual problem, but also rule out that the mod plays foul.
User avatar
Graf Zahl
 
Joined: 19 Jul 2003
Location: Germany

Re: The importance of simple examples in bug reports

Postby Xaser » Wed Mar 15, 2017 10:55 am

I think this announcement focuses a bit too much on the case where the person who's filing the bug report is also a modder. I'd wager that's a pretty big percentage, but it lacks directions for the folks who wouldn't know how to put together an example wad.

For the lay-players out there, perhaps we should list the following bits of advice:
  • Exact version numbers are pretty much mandatory.
  • If the issue is a crash or something graphics related, OS and hardware should be listed.
  • List exact steps to reproduce. Things like "Optionally, play some of the missions" aren't any good. :P
  • Savegames are handy.
  • If all else fails, consider notifying the person who made the mod, as they may be better equipped to debug it.
    • Same advice applies to the devs -- if debugging a mod issue is going to be troublesome, roping in the modder may ease the burden.

If it's possible to cut out complex mods from the equation, that's best, but it's just not always possible. I just don't want to start seeing reports closed because "mod 'X' is too big" and no other reason -- "On Hold", sure though. ;)
User avatar
Xaser
autodefenestrator
 
Joined: 20 Jul 2003


Return to Developer Blog

Who is online

Users browsing this forum: No registered users and 1 guest