Automated Tests and Human QA: A Love Story

A little background for our series. We know that every agile development team needs to automate testing. Continuous integration systems that repeatedly check known test cases -- unit tests, integration tests, API tests, and often functional tests -- are necessary. But not sufficient.

For example, a computer can click a button that a human can’t see because it’s blocked by something else on his phone. If that button says, “Buy Now,” you’re losing money.

Humans discover. They know how to find the kinds of bugs that development teams and automated tests miss. In particular, they excel at finding:

• Unforeseen problems, where human testers’ fresh eyes and drive to discover complement automated checking by developers’ tests.
• Usability issues, which may be “to spec” but which drive customers away.
• Real world issues involving phones, cameras, IOT devices, payment systems, and settings that are difficult to simulate in the lab.

In short, they discover precisely the kinds of issues that make the difference between a satisfied customer who leaves a five-star review and an angry customer who leaves a one-star review.

Believe it or not, sometimes people can’t log into apps that have passed all of their automated tests.

Since Pokémon is a thing these days, we’ll start with them. Here the poor kid (OK, it’s probably an adult) sees “Failed to log in” and “Sign Out” on the same screen, plus one button overlapping another.

How are you going to catch them all if you even can’t log in?

It’s hard to know what was actually happening in the Pokémon app. Sometimes it has to do with phone permissions, or with a dependence on a clear signal.  But with other apps it’s more obvious. Take a look at this one.

Here, what’s separating you from your donut is your gigantic fingers, which can’t activate the input field on the form. Eating fewer donuts might be a good idea for other reasons, but might not help you get into this form.

So how does this kind of thing get through automated tests? A couple of hypotheses: even though the form field isn’t possible for a human to access, the field has been rendered, so an automated test looking for that field will find it and fill it out. Alternatively, perhaps this phone’s screen is uniquely sized to cause this problem to happen, and the automated tests were only working against one or two screen sizes.

In a mobile-first world, you’d be surprised (or maybe you wouldn’t!) how often companies neglect testing on actual mobile devices.

This concludes the first edition of “bugs people find.” Stay tuned to the blog and we’ll keep you informed with fascinating (okay, if you’re into that) tales of bugs our testers uncover.