Crowdsourced Testing: How to Succeed, Insights from Martin Reinbach

Sean: You work quite closely with our customers, especially when onboarding them. What surprises them at the beginning?

Martin: Most customers are pleasantly surprised at how easy it is to set up a test cycle. Many of our customers use our service in conjunction with in-house automated tests, which take more time and effort to create and have to be precisely specified, so they’re hard to change as your project evolves. By comparison, our crowdtesting approach is quick, flexible and agile -- a feature that is key to helping companies meet the demands of continuous delivery.

Sean: You mention continuous delivery. Could you explain to me how the software development process changes as a company starts to use crowdtesting as part of their testing protocol?

Martin: Most internal QA teams face two challenges: limited resources (devices, people, time), and confirmation bias. Companies that partner with us are able to eliminate both problems. Regarding resources, our testers provide every possible device, browser, and software, and they test concurrently so personnel bottlenecks are not a factor. So our customers are able to gain a more thorough understanding of the state of their software earlier in the development process than they used to, and as a result they can plan better.

Regarding bias, because our testers experience and interact with the app without preconceptions, like their customers do, they don’t suffer from the same biases that an in-house team would; they’re able to offer not only functionality feedback, but also usability suggestions -- again, earlier in the process.

By decreasing the time between development and release and pushing feedback earlier in the cycle, teams become more agile and able to approach continuous delivery.

Sean: Speaking of internal QA teams, can you talk a bit about how the in-house teams work with our team of crowdtesters?

Martin: Obviously, our crowdtesters are able to take away the hassle of manual functional testing, cross-browser testing, and iOS and Android testing. They provide a comprehensive set of results featuring screenshots, video recordings, and crash reports, which can be delivered rapidly. This means that bugs are detected and fixed faster, keeping apps and websites from getting bad reviews (ultimately aiding in customer retention and preventing revenue loss).

Less obviously, QA is fundamentally about managing risk. Two other major areas where our testers work with in-house teams to mitigate business risk are through their usability suggestions and the release readiness score. Feedback on the ways in which people engage with your app before it goes live is irreplaceable. It essentially gives the company a vote of confidence before it is released to the general public.

Sean: What can companies do though to make sure that they get the results they’re looking for?

Martin: The most important thing is to know the purpose of your test. We have customers who test very early in their development process, even at the wireframe stage, to get usability feedback; we have others who do exploratory tests as soon as they have something functional to get insights into where the hotspots are; we have others who test before a code merge to make sure everything seems right before they make a big change. All of these are great uses of the crowd! You just need to make sure the testers understand what you're looking for and why you're running the test.

Sean: Any last tips, tricks or advice?

Martin: For a big release, with a lot of changes and a need for thorough testing, run the test longer. The crowd works over the weekend even if you don't, so if you have the opportunity to run a big test over the weekend, take it.