User Acceptance Testing: Better, Faster, Stronger

Traditionally, a major challenge of user acceptance testing is the need for actual users to interact with your software. By providing users with access, they see all the bad (along with the good) that your product has to offer. Often, this forces the development team to postpone UAT until fairly late into the software development life cycle, since it would be tragic to open things up to your customers in UAT only to have the software crash and burn in the process.

Terrible public relations aside, the pain of waiting to begin UAT -- which is often self-inflicted by developers, managers, or marketers -- means that the software itself suffers for the lengthy period leading up to when UAT can officially begin. During this time, user acceptance testing is often ignored, which can cause major problems and delays later in the development life cycle as unforeseen issues emerge. Typically these problems might include issues with the UI, localization, connectivity, system integrations, and many other end-to-end functionalities that weren't fully tested up to this point.

Even in the best-case scenarios where your budget allows for the employment of a team that is able to perform UAT right from the get-go, rarely can an organization maintain a roster of testers -- or customers -- large enough or diverse enough to handle every problem or foresee every potential issue. Giants like Facebook and Google, notwithstanding, most organizations can’t easily show their new builds to a million customers to see what happens.

Stirring in a crowdtesting solution into the QA-mix is an extremely powerful tool, as it provides immediate user acceptance testing capabilities, along with many additional benefits:

• Crowdtesters can mirror your intended audience.
• You can show crowdtesters pre-release software without negative consequences.
• It's easy to manage crowdtesters.
• You can show crowdtesters big or small chunks of software, so you can do it as often as you need to.
• Unlike your friends or employees, crowdtesters have no reason to lie to you about how good your software is.

By assigning a veritable army of professional testers to your project via crowdtesting, all manner of tests can be performed, covering the entire spectrum of test cases your software requires, including those pertinent to user acceptance testing. Best of all, crowdtesting ensures that your project can "turn on" UAT exactly when and where it's necessary, rather than waiting until late into the development life cycle, when a traditional beta phase might begin.

Crowdtesting brings another major benefit to the user acceptance testing process: the ability to perform not only "traditional" software testing, but also the capability of assigning more involved tasks and test cases to crowdtesters, which suit the specific needs of your team and your project.

For example, testers might be assigned an end-to-end test case to cover everything from registering and logging in as a new user to adding items to the cart and making a purchase. While the success-rate of this task alone is useful data, the sheer quantity of testers involved in crowdtesting means that you can gather far more information than the basic test case would ask of them:

• Did the test perform as expected?
• On a sliding scale, how difficult was the process?
• Were any elements in the application confusing or requiring more clarification?
• Which step(s) gave you the most/least trouble?

The possible lines of inquiry are as numerous as your software is complex. What's critical is that crowdtesting provides a huge surge of unique individuals who can test and examine your application in virtually any manner you require, ensuring that user acceptance testing is brought to the forefront of the development process from the very outset.

Numerous usability and acceptance testing methodologies are commonly used in modern software development. At one end of the spectrum sits a method like hallway testing, which selects an assortment of people to test your software. As the name suggests, these people could be pulled from the hallway outside your office door. Typically, hallway testing is best for situations where test cases are broad, and the system simply needs to be checked for any major flaws that prevent basic, fundamental functionality from operating properly.

On the other end of the spectrum of testing methodologies sits usability audits, which typically requires an intensive evaluation of the software interface and usability. Ideally, this is overseen by usability experts with a representative sample of end-users, in stark contrast to the laissez-faire selection methods of hallway testing.

Nestled nicely in between these two divergent methods of usability testing sits crowdtesting, which combines many of the benefits of both hallway testing and usability audits.

Due to the volume of individual testers that can be assigned to your project, crowdtesting provides a much larger user base for testing purposes than the hallway testing method (and likely more people than usability audits, as well). Since testers in a crowdtest environment are available from many locales all around the globe, crowdtesting presents a far more diverse user base than traditional usability testing methods. Moreover, crowdtesters won't share the same cognitive biases that in-house testers or other hallway denizens from your own organization might possess.

While crowdtesting doesn't purport to be as intensive a study as a full-blown usability audit -- which typically includes a deep analysis of both the software and the testers -- crowdtesting does provide many of the benefits of usability audits, often with significantly reduced cost and certainly reduced manpower from your organization. As professional testers, crowdtesters can be asked to evaluate almost any aspect of the software, from common fault identification and reporting to elaborate usability test cases, as we saw in the previous section. Crowdtesting offers 80% of the value of a usability audit, at a fraction of the time and cost, by supplying diverse people who can use your software without the need to tell them exactly what to do, who can then give you some level of structured feedback in return.