Best Practices in Managing Crowdtesters

One of the foundations of crowdtesting is that testing should take place under real-world conditions and not in a testing lab or office. To get this natural user mindset and behavior, though, it's important to set up conditions that allow and even mandate this. For example, taking the approach of exploratory testing makes it much more likely for a tester to approximate natural user behavior. The tester's mission is to explore the entire app or website without strictly predefined guidelines.

Other commonly deployed software testing methods don't offer or strive for this kind of view on checking software. In scripted testing, for example, testers are usually given step-by-step instructions to achieve certain results. On the one hand, this generates more predictable results, but on the other hand you get far fewer insights into user behavior and expectations.

Recruiting a representative and professional tester group is crucial to crowdtesting success. However, we don't take the motivation of each individual tester for granted. We make sure each tester is incentivized to find and report great bugs, primarily by offering bonuses tied to the type and severity of the bugs found.

We've found compensating crowdsourced software testers based on these concrete and defined goals to be most effective. By doing so, we make sure that relevant bugs with thorough documentation get submitted, so that only reproducible bugs are submitted to developers. This also makes it clear to testers what they should be aiming for.

By rewarding our testers monetarily and only after customers approve their submitted bugs, we've built in the right feedback mechanisms to ensure quality and performance .

While money is one lever for maintaining motivation in testers, other kinds of rewards and incentives also help get participants in crowd projects excited. While testers may initially be drawn to testing cycles because of the financial compensation, over the long term, it's smart to build in other kinds of recognition and feedback to keep them motivated. We've built in mechanisms like leaderboards, points, and badges to make sure testers who perform exceptionally to get recognized.

This is, of course, tied in with active tester community management. In order for these kinds of community recognition incentives to succeed, testers need to feel like they're part of a community whose recognition and acclaim they want. A well-managed community with rankings for different dimensions of successful bug hunting, a point-based system that issues points for specific kinds of performance, or badges that recognize expertise or particular engagement (like iOS testing or nightshift testing), or some combination of the three, has the potential to generate extraordinary outcomes.

These three practices enable us at test IO to get the most out of our tester community and to provide the right incentives and structures for both customers and testers to succeed.