How to Prepare Software for Crowdtesting

Crowdtesting is an excellent way to help ensure that end-users have a high-quality experience with your software. But, it isn't necessarily easy to have your engineering team start working with crowdtests.

Over the last decade, we’ve worked with numerous teams and companies to bring crowdtesting into their processes. We have found that there are five basic tips that can ease the transition. Here they are:

A "staging environment" is an environment where you can deploy software, which is as identical to your production environment as possible—but without actually being your production environment. This will allow you to run tests with an external crowd and actually catch bugs before paying customers or end-users encounter them.

In the jargon of software, testing a "fixture" is a predefined set of records that can be immediately written to your database. In order for testers to meaningfully test your software, the system needs realistic data that mimics production. Here’s the caveat; you don't actually want any real production data exposed to external testers. This is why you need fixtures.

Essentially, you prepare a snapshot of your database so that it has all of the data and records necessary for your system to be up and running for testers — think of user accounts, product records, etc. Then, when you deploy your latest version to your staging environment for testing, you simply apply this "snapshot" to your staging database.

In order for testers to have access to your system, it’s likely they will need to log in. In order to log in, they will need credentials. Building on the general need for having data prepared to set up your database to look production-like, you specifically need a large set of user accounts to provide to the testers. We recommend a large set so that testers don't accidentally "step on each other’s toes" while testing. You don't want a bug reported that a product disappeared from the cart simply because two testers signed in with the same credentials and were working out of the same cart.

Crowdtests are excellent for exposing bugs that are more advanced in complexity, not simple problems like performance ineptitude; your software thought it could do X, but in fact it couldn't. In fact, many bugs are a version of this problem. When software tries to perform a function it isn’t capable of, it tends to be pretty noisy. This is where error monitoring services come in.

When you connect an error monitoring service to your application, you start to capture these lower-level issues, along with a wealth of extra information and context. Since many of the bugs reported by testers relate to some loggable error, you will save your developers hours or days of tracking down the source of the bug. With this, your developers can simply find the error that connects to the bug report. Then they have the full suite of low-level information that allows them to find out where this nasty little bug resides, down to the exact place in the codebase.

Finally, it is important that the descriptions you have of how your software is expected to work are always up-to-date. This is foundational to running top-notch crowdtests, and these feature descriptions provide the context and expectations for testers. It is a waste of both testers’ and developers’ time if bugs are reported because something was said to be expected but is no longer a part of some particular feature. Respect the time and value of the testers reporting bugs and the developers fixing them. Invest the time in explaining how things should work as clearly and candidly as possible.

Implementing crowdtesting into your software processes involves planning and consideration. With these suggestions, you can guide your company into a smoother transition.

To learn more about crowdtesting, check out the test IO blog.