What Makes a Good Bug Report, and Why Your Users Are So Bad at Providing Them

I must confess that I am still continually surprised at how often I get poor bug reports. After still another partial bug report that requires me to write back with a collection of questions to gather enough information for me to begin investigating, I thought it was time to sit down and actually think through why there is so often such a gap between what information users provide when reporting a bug and what information developers want when receiving a bug report. After a good think, I've come to this conclusion: A good bug report requires context, and this is precisely what users are least likely to be able to provide. 

What, you might be asking, is the context that is required for a good bug report? Permit me to respond with a question of my own. What one piece of information could be added to this screenshot to dramatically increase its value as a bug report attachment? 

This is a standard example of the kinds of attachments I get in bug reports, and, while I appreciate the time and care taken to highlight the key points in the image to consider, I would appreciate it so much more if they had included the URL shown in the URL bar at the top of their browser. This is a vital piece of context required for me to diagnose the source of this data mismatch. 

Indeed, this screenshot—and all bug report attachment screenshots—would be immediately improved if the user hadn’t cropped things at all. Take a screenshot of your entire window. Show me the URL, show me the navigation of the webapp, show me everything. Why? Because before I can figure out what the problem is, I have to first determine where the problem is.  

Bug reports from users are so often overly focused on showing what the problem is that they provide me no way to determine where the problem is. Remember, in order for a developer to fix a bug, they have to find a path from the content on your screen to the relevant files in their codebase. And any codebase driving a software product that users find valuable enough to pay for will have hundreds if not thousands of files. And if that screenshot above came from a user that I could not ask for a larger screenshot of the entire window, the only way I could trace my way to some relevant files would be to search for every occurrence of “Known bugs” in the entire codebase and start checking each file to see if was responsible for this exact snippet of a page. This is tedious work, made even more so by the knowledge that none of this would be required if the bug report had simply included or shown the URL. 

So, I need the context of where you are when you encounter the bug. But I will also need the context of how you got there. What exact steps did you take to end up in this buggy situation—which links did you click, which form fields did you fill out, which buttons did you press? Bugs, like the software they hide within, are not static. They occur not just in particular places but under particular conditions. In order to hunt them down and fix them, I need to find both the place they are hiding but also what conditions draw them out and onto your screen. This means that a good bug report will include the steps taken that lead to the bug experienced. It would also contain notable aspects of the state of the system when the bug was encountered. If you are reporting a bug found in the “Shopping Cart” page of an e-commerce site, it is important to note if you added the product to the shopping cart from a product page, from a search results page, or from your wishlist. Had you just updated the quantity? Did you accidentally remove it and then add it back? Providing this kind of information is the difference between a poor and an excellent bug report. 

But, if a great bug report is primarily about providing context, why is that developers everywhere are still getting so many poor bug reports? The answer is, sadly, because context is precisely what users are unlikely to provide. In order to provide the exact series of steps they took leading up to finding the bug, users would have to be paying attention to each step they took and cataloguing them in their minds. In order to tell you some notable condition they were in when they found the bug, they would have to be paying attention to what they were doing and what conditions they were applying to the system. In order to think showing the URL is important when reporting a bug, they would have to know what a URL is and how it is important in web applications. The simple fact of the matter is that users won’t simply think of these things. They are using your application, not hunting for bugs. They stumble upon bugs, haphazardly and surprised. They aren’t paying attention to the steps they are taking, to the URL of the page they are on, or what conditions the system is in; they are simply trying to use your software to do something useful for them. That is to say, they can’t provide you the context for the bug because they aren’t paying any attention to the context, only to their task. 

So, where does that leave us? Are we, as developers, doomed to only ever receive poor and mediocre bug reports? Perhaps, but not necessarily. Because it is possible to engage users to use your software, not to accomplish their task, but to hunt for bugs. If you want good bug reports, you need to have users use your software for the express purpose of finding bugs. Then they will be paying attention to context, they will know how to articulate and structure their findings, and they won’t be upset when they find a bug. This is why I work for a crowd-testing platform, because I believe that developers everywhere deserve high-quality bug reports, but I know that they aren’t ever going to get them from their every-day users.