A test IO Bug's Life

You click submit on a new crowdtest, finish up your day and go home. When you wake, detailed bug reports are waiting in your Jira and you immediately get to work improving your app. It seems effortless, but behind the scenes, a lot of steps go into making sure that every crowdtest delivers valuable results. This is the story of how those bugs get from tester to you.

Test Submission

It starts when you first create a test. Your requirements determine the kind of bugs that testers find and report, be it functional, usability related, or content and visual. That is why we do not immediately send the request to the crowd when you click submit. Instead, the details of the crowdtest go to a group of people known as Team Leads. Team Leads are some of our most veteran testers and are responsible for making sure that each test is up to test IO standards. To do that they perform many functions. The first is ensuring that the test instructions are clear and that the given environment can be accessed. The tests that meet these requirements are "accepted" and passed along to the testers. We wouldn't want to spend time on something that can't be tested, right?

Bug reporting

With the test accepted, invites go out to the testers that fit your device (and other) requirements. About 50 will join the test and start exploring your website or app. When they find a bug they will submit a bug report. Those reports must include:

• A detailed bug title
• The URL of where the bug was found
• Plain English steps to reproduce the bug
• The expected vs the actual result
• A screencast and/or screenshots of the entire interaction
• A suggested severity

Then the bugs are sent to you, right? Not yet! Each bug must be individually reviewed before reaching your bug tracker. Our software takes the first stab. The test IO platform will flag any bug titles that are similar to any other bugs found during the test cycle or in the known bug list. If the bug was already found by another tester, the device, OS, and browser details, will be added to the preexisting bug report. If it is known the bug is removed from the list. If it is unique it will move on to the final human checkpoint, the Team Lead.

Final Review

Team Leads have been exposed to every testing trick in the book. That is why we rely on their eyes to review each bug. They check them for three things: Quality, Validity, and Importance.

Quality - Does each bug contain the requisite info? Is the screencast clear?

Validity - Can I reproduce the bug? Is the bug within the scope of this test?

Importance - Is the severity of the bug correct? Will the customer find the bug relevant?

This is the last check a bug goes through, but that does not mean it only goes through it once. If a bug is missing details a Team Lead can request those details directly from the tester. The tester then has a chance to edit their bug report and resubmit it. This helps make sure that no good bugs slip through the cracks, because while it might be missing a screencast it may be the critical issue that makes the test. Finally, the bugs that make it through this last round of checks are forwarded to you!

Every forwarded bug goes through this process. That's why test IO is confident that every crowdtest run will give you results that provide value immediately. If you would like to see what kinds of bugs the crowd can find on your product, click here.