Crowdsourced Testing Tips and Tricks With Matt James

Originally from the United Kingdom, Matt James  has been working with test IO for a year, bringing his expertise as we expand into the US and Canada. In addition to working with our North American clients, he works with our testers to make sure they’re happy and our customers are successful. I recently sat down with Matt to learn why our customers are so eager for his advice.

Sean: What’s the biggest surprise for our customers after seeing our initial test?

Matt: Our customers are surprised at where our testers find bugs. Often times, our testers are able to uncover bugs that internal teams didn’t consider.

Sean: What kinds of bugs do clients expect? Which ones are surprising?

Matt: Customers often come to us expecting their product to be ready for release, so it often comes as a shock when our crowd testers find critical bugs. And often times, bugs are surprising because they pop up in areas that their internal team wasn’t expecting. That’s the advantage of exploratory testing -- that it reflects the way people would actually engage with the app as a whole, and not simply with a single feature.

Sean: What kinds of differences do customers see between the testing done by crowdtesters and testing performed by their internal QA teams?

Matt: The primary difference is the ways in which our testers approach the app. Although clients can be a bit hesitant at first, they realize that without a prior knowledge of the website or app, crowdtesters engage with the app more organically than their internal teams, so they mirror real-user behaviors. Because QA teams are often limited in their time and resources, internal QA teams can struggle to take a step back from the website or app, and are blind to some of the errors that our testers find.

Sean: That’s interesting! How else do software development processes change when a company takes full advantage of crowd testing?

Matt: Quite simply, those companies who fully embrace crowd-testing are able to move faster. They can run small scope tests on any new feature whenever they wish and if they have another feature the next day, they can do the same process again. Handing over some of the exploratory testing gives back time to the developers to get on with fixing bugs and developing features. One thing we say is that once you understand what the crowd can do is that it’s like going from a landline to a cell phone: you find a bunch of new ways to use the crowd that you wouldn’t have imagined.

Sean: What can a company do to get good results and uncover the bugs most relevant to them?

Matt: I think the key thing is to imagine yourself as one of the testers. It is very easy to forget that not everyone will have previous knowledge of a particular app or website so providing good, descriptive information when setting up the test really pays off. The same is true when defining the things you don’t want -- we all know that we are not always interested in every possible type of bug, but a tester has no way of knowing that unless they are told in the description.

Sean: Are there any other particular pitfalls companies make when starting with crowd testing? What can they do to avoid them?

Matt: I coach customers to avoid three pitfalls:

Trying to script the test: Although it is important to offer guidance, you will get more value when you free the testers to learn and engage with your product. We’ve learned repeatedly that this produces the most robust and insightful results.

Not providing enough information: Knowing how much information to provide is a double-edged sword. Too much detail or scripting of the test inhibits the ability of a tester to give a comprehensive assessment of the app. But not providing enough information is also dangerous, as it makes it impossible to assess what is valuable. When scoping, it’s important to explain the feature and its expected functionality and not define a set of steps.

Limiting their testing to what they’ve previously done: Because most companies struggle with limited time and resources, they often have to limit their testing. When companies start working with test IO, they sometimes forget that by hiring us they can now begin more robust and holistic testing of their software. Test IO has a bank of 20,000+ testers that can handle any product, software, or browser. Get out and explore; take advantage of the capabilities that this new technology offers!