Automated Tests and Human QA: A Love Story

August 3, 2016

A little background for our series. We know that every agile development team needs to automate testing. Continuous integration systems that repeatedly check known test cases -- unit tests, integration tests, API tests, and often functional tests -- are necessary. But not sufficient.

When the customer experience really matters, people matter.

For example, a computer can click a button that a human can’t see because it’s blocked by something else on his phone. If that button says, “Buy Now,” you’re losing money.

Humans discover. They know how to find the kinds of bugs that development teams and automated tests miss. In particular, they excel at finding:

  • Unforeseen problems, where human testers’ fresh eyes and drive to discover complement automated checking by developers’ tests.
  • Usability issues, which may be “to spec” but which drive customers away.
  • Real world issues involving phones, cameras, IOT devices, payment systems, and settings that are difficult to simulate in the lab.

In short, they discover precisely the kinds of issues that make the difference between a satisfied customer who leaves a five-star review and an angry customer who leaves a one-star review.

I can’t @#(%(@ log in!

Believe it or not, sometimes people can’t log into apps that have passed all of their automated tests.

Since Pokémon is a thing these days, we’ll start with them. Here the poor kid (OK, it’s probably an adult) sees “Failed to log in” and “Sign Out” on the same screen, plus one button overlapping another.

How are you going to catch them all if you even can’t log in?

It’s hard to know what was actually happening in the Pokémon app. Sometimes it has to do with phone permissions, or with a dependence on a clear signal.  But with other apps it’s more obvious. Take a look at this one.

Here, what’s separating you from your donut is your gigantic fingers, which can’t activate the input field on the form. Eating fewer donuts might be a good idea for other reasons, but might not help you get into this form.

So how does this kind of thing get through automated tests? A couple of hypotheses: even though the form field isn’t possible for a human to access, the field has been rendered, so an automated test looking for that field will find it and fill it out. Alternatively, perhaps this phone’s screen is uniquely sized to cause this problem to happen, and the automated tests were only working against one or two screen sizes.

In a mobile-first world, you’d be surprised (or maybe you wouldn’t!) how often companies neglect testing on actual mobile devices.

This concludes the first edition of “bugs people find.” Stay tuned to the blog and we’ll keep you informed with fascinating (okay, if you’re into that) tales of bugs our testers uncover.

Read More

June 13, 2019
The Important of Manual-Stage Testing

Incorporating manual testing into your development pipeline, without bottlenecks.

June 7, 2019
Regression Testing: Where Does Crowdtesting Fit In?

What is regression testing, and how does crowdtesting benefit it?

May 30, 2019
HUM Nutrition: A Test a Day Keeps the Bugs Away

test IO helps HUM Nutrition provide customers with life-changing products.

Ship Faster, Sleep Better

Get a Demo
testIO Logo Footer
test IO Inc.
1111 Broadway, Floor 3 
Oakland, CA 94607

test IO GmbH
Sonnenallee 223 a
12059 Berlin