What is Crowdtesting?

March 1, 2017
Powen Shiah
Powen Shiah

Crowdtesting (or crowdsourced testing), is a practice that harnesses the efficiency and advantages of crowdsourcing for software testing. Software such as mobile apps, SaaS, native desktop applications, or websites in various stages of development, is distributed to a large group of people. The people in this group, the crowdtesters, run the software on their own computers or mobile phones to inspect it for defects and areas for potential improvement. At test IO, these crowdtesters don’t need to be given specific scripts; they’re usually asked to evaluate specific sections or features of the application, using their experience and best judgement to unearth and identify problems. Reports from the testers flow back to the software team, at test IO typically after being examined by a senior tester.

Product, development, and QA teams can use the crowd for regression testing, feature testing, usability testing, user acceptance testing, cross-browser testing, routine sanity testing, and more. The power of crowdtesting comes from being able to draw from large pools of testers, comprising people with different skills, inclinations, experience, and devices. Crowdtesting thus combines human testers with broad hardware coverage at scale. Harnessing human intelligence in groups gives even fast-moving development teams real-world testing capabilities that wouldn’t have been possible a few years ago.

Human Diversity

Software testing benefits from the diversity of the crowd. It ensures a wide range of attributes and characteristics that in-house QA teams typically can’t match, including:

  • Locales and languages
  • Levels of expertise
  • Demographic characteristics
  • Life experience and affinities

By making sure a diverse group of testers evaluates a project, a QA team can feel confident that their software has been thoroughly evaluated by people who were not part of creating it. This means the testers do not have the embedded biases and expectations testers from within the company would. Furthermore, including testers from different backgrounds, countries, and ages as well as other characteristics helps to catch any other assumptions that may be built into the product.

Crowdtesting also gives companies the chance to try out their products on testers similar to their target customers. With a big enough pool of potential testers to draw on, it’s possible to set up a test with people who closely track these characteristics, whether it’s age, country of origin, or another attribute. By assembling crowdtesters who are, for example, Facebook power users, train commuters, or enthusiastic video gamers, a product team can test on a group like their customers without actually testing on their customers.

With a global testing workforce, a company in New Zealand can see whether the app they’ve developed also works in Chile, Japan, or Italy. By bringing in crowdsourced testers in the prototyping phase, user experience researchers can avoid replicating their own biases into their studies.

Hardware & Environmental Diversity

In addition to the broad spectrum of people, crowdtesting also means that software can be tested on huge variety of devices in many different real-world environments. The more testers in the crowd, the more diverse the devices they’re using to run your software will be. This includes variations in form factor, hardware, and the operating systems. Even different versions of the same OS present some of the greatest development challenges – Android is a prime example.

Crowdtesters don’t simply follow scripts the way an automated test would; they use their human discretion and accumulated knowledge to discover functional and user experience flaws in products and platforms. Testers use their devices as real users do, with privacy settings, ad blockers, and locales configured to provide realistic scenarios unattainable in a lab setting. With dozens or hundreds of testers, it’s human exploratory testing in highly varied real world conditions at unprecedented scale and efficiency.

Efficiency & Scalability

Testing at scale on a multitude of hardware and software combinations doesn’t have to be limited to big consumer software corporations who try out new code directly on customers. Crowdtesting enables smaller or non-consumer-facing software companies to test their software at scale in real-world conditions. This means they’re better prepared and have less to worry about when releasing to wide audiences.

Crowdtesting adds both flexibility and extra testing resources when software teams need them most. Setting up regular crowdtests means QA can keep up with the continuous development cycle. When a milestone approaches, testing doesn’t have to fall by the wayside or get pushed to the end. Crowdtesting is available on-demand and doesn’t redirect development resources. Right before a big release, crowdtesting is flexible, so you can get dozens or hundreds of testers for a final test cycle without pulling engineers off of development.

Running tests in parallel is another efficiency benefit of crowdtesting. Instead of having one or two testers run through the scenarios and new features in a piece of software on a single device, multiple crowdtesters on different hardware and software can test all the scenarios at the same time. This gets you results faster, and helps to pinpoint problems in certain builds or hardware versions.

Finally, crowdtesting at test IO simplifies the QA team’s workflow and eases the burden of testing. You can work without detailed test cases: explain what you want in plain language. Our crowdtesters will understand without needing step-by-step scripts. Even if a problem does crop up that would stop an automated test, humans are robust. Human crowdtesters can recognize that mistakes or misunderstanding and keep on testing.

Crowdtesting at test IO

Unlike other crowdtesting providers, test IO has – over the past five years – learned to differentiate testers and augment their skills. By selecting capable testers and differentiating them over time, we concentrate on bringing a highly-qualified crowd to the table. Our testers often already work as software professionals, and they’re looking to expand their skill sets or to do challenging and rewarding work on the side.

All this results in a professional testing workforce you can use to continuously test your software, that’s flexible enough to add testers when you need them, and experienced enough to know what you need without scripts or step-by-step instructions.

Read More

Ship Faster, Sleep Better

Get a Demo