How to get the most from your crowdtests? Tips from our CSMs

With crowdtesting, companies can have real people test their software in real-world conditions. This is a great way to find out how their products work in the hands of real users. But while running a crowdtest can be as simple as putting your product in the hands of testers, it takes more than that to get the best results. So I sat down with our CSMs to find out what it takes to get the most from a crowdtest. These are a few of the things that stood out.

Our testers are experienced, but that does not mean they will understand your product's nuances. Testers don't know your product as well as you do. People come to us because they want feedback from an end-users perspective to catch bugs that a typical consumer might find, which would be way more costly. Feature descriptions are the only way for testers to know how to navigate your product in search of issues relevant to you. Otherwise, it's complete black-box testing. So, when writing out test instructions, do not use jargon and internal naming conventions. Use language that a customer who is seeing your product for the first time could understand.

Yes, testers get paid to test apps, but that does not mean they need to test your product. The community is like a marketplace, and our testers like challenges, but not impossible tasks. That means we have to make tests, and coming back to tests, as easy and attractive to testers as possible.

Put yourself in their shoes. You log onto the platform and start reading the test description for a fintech application. It is an exploratory test, but it uses internal financial jargon, is unstructured, is an information dump, has unclear user flows, and has gaps in expected behavior. This would be unattractive to testers. This does not mean they cannot test complex applications. We've tested apps ranging from crypto trading, streaming, and payment processing. What's made them testable is clear, concise instructions and user flows that make it easy for testers to understand and explore. For further details, check out this article on exploratory testing vs. test case testing. 

Exploratory tests require a lot less effort to maintain than test cases, but that does not mean they do not need to be maintained. Just like your software evolves, so do your tests. So, when you change the functionality of your software product, it is important to reflect that in the test IO platform. In our type of testing, we only have Feature descriptions, essentially the set of core functionalities of that Feature to be checked. What we're missing is documentation/input on the test build releases to provide more context for new Features or new functionalities for old Features. Updating Features with that context allows for more efficient testing as it enables our testers to focus on the parts of the product that are dependent on the changed code. It also stops testers from submitting bugs for things that are intended functionality, but do not match the feature description.

Feedback goes a long way toward improving your bug results. Some of the best pieces of feedback you can give them are standardized within the platform. You can accept bugs, reject bugs (and specify the reason), and mark them as known. This information improves your test cycles at every turn. When you repeatedly accept a bug type, we'll make sure that any similar bugs will make it through the initial review. If you reject a kind of bug consistently, then we'll work with you on fine-tuning your tests so you'll stop seeing those kinds of bugs in future tests. Finally, if it's in your known bug list, testers will know not to report the issue, so it has no chance of making it past review.

From these tips, a few things are clear. It is best to treat your testers like an extension of your QA team, human but external. Give them information that is easy to understand and provide them with feedback when they make mistakes. No one is perfect and especially not on their first try. But if you work with our testers, they will reward you with stellar test results that will help you make your software better than ever.

If you want to start testing your software in real-world conditions, reach out here.