Automated Tests and Human QA: A Love Story

August 3, 2016
Phil
Phil

A little background for our series. We know that every agile development team needs to automate testing. Continuous integration systems that repeatedly check known test cases -- unit tests, integration tests, API tests, and often functional tests -- are necessary. But not sufficient.

When the customer experience really matters, people matter.

For example, a computer can click a button that a human can’t see because it’s blocked by something else on his phone. If that button says, “Buy Now,” you’re losing money.

Humans discover. They know how to find the kinds of bugs that development teams and automated tests miss. In particular, they excel at finding:

  • Unforeseen problems, where human testers’ fresh eyes and drive to discover complement automated checking by developers’ tests.
  • Usability issues, which may be “to spec” but which drive customers away.
  • Real world issues involving phones, cameras, IOT devices, payment systems, and settings that are difficult to simulate in the lab.

In short, they discover precisely the kinds of issues that make the difference between a satisfied customer who leaves a five-star review and an angry customer who leaves a one-star review.

I can’t @#(%(@ log in!

Believe it or not, sometimes people can’t log into apps that have passed all of their automated tests.

Since Pokémon is a thing these days, we’ll start with them. Here the poor kid (OK, it’s probably an adult) sees “Failed to log in” and “Sign Out” on the same screen, plus one button overlapping another.

How are you going to catch them all if you even can’t log in?

It’s hard to know what was actually happening in the Pokémon app. Sometimes it has to do with phone permissions, or with a dependence on a clear signal.  But with other apps it’s more obvious. Take a look at this one.

Here, what’s separating you from your donut is your gigantic fingers, which can’t activate the input field on the form. Eating fewer donuts might be a good idea for other reasons, but might not help you get into this form.

So how does this kind of thing get through automated tests? A couple of hypotheses: even though the form field isn’t possible for a human to access, the field has been rendered, so an automated test looking for that field will find it and fill it out. Alternatively, perhaps this phone’s screen is uniquely sized to cause this problem to happen, and the automated tests were only working against one or two screen sizes.

In a mobile-first world, you’d be surprised (or maybe you wouldn’t!) how often companies neglect testing on actual mobile devices.

This concludes the first edition of “bugs people find.” Stay tuned to the blog and we’ll keep you informed with fascinating (okay, if you’re into that) tales of bugs our testers uncover.

Read More

April 19, 2019
Will Automated Testing Ever Replace Manual Testing?

Will manual testing stay relevant, or are the machines taking over?

April 18, 2019
The Downfall of Manual Testing

Is manual testing being left in the dust?

April 15, 2019
Customer Conversation: Finding Your QA Fit

Looking to learn more about setting internal quality standards and finding impactful bugs? Hear what our veteran customers had to say!

Ship Faster, Sleep Better

Get a Demo
Follow us on:
twitterfacebooklinkedin