Automated Tests and Human QA: A Love Story
People sometimes ask us, “What kind of bugs do your testers find even when there’s a lot of test automation already?” Today we inaugurate an occasional series that highlights interesting bugs (and types of bugs) that our testers find. Don’t worry, customers, we won’t share your data. The truth is, a lot of the things we find before release are out there in production software every day! We can use those.
A little background for our series. We know that every agile development team needs to automate testing. Continuous integration systems that repeatedly check known test cases — unit tests, integration tests, API tests, and often functional tests — are necessary. But not sufficient.
For example, a computer can click a button that a human can’t see because it’s blocked by something else on his phone. If that button says, “Buy Now,” you’re losing money.
Humans discover. They know how to find the kinds of bugs that development teams and automated tests miss. In particular, they excel at finding:
In short, they discover precisely the kinds of issues that make the difference between a satisfied customer who leaves a five-star review and an angry customer who leaves a one-star review.
Believe it or not, sometimes people can’t log into apps that have passed all of their automated tests.
Since Pokémon is a thing these days, we’ll start with them. Here the poor kid (OK, it’s probably an adult) sees “Failed to log in” and “Sign Out” on the same screen, plus one button overlapping another.
How are you going to catch them all if you even can’t log in?
It’s hard to know what was actually happening in the Pokémon app. Sometimes it has to do with phone permissions, or with a dependence on a clear signal. But with other apps it’s more obvious. Take a look at this one.
Here, what’s separating you from your donut is your gigantic fingers, which can’t activate the input field on the form. Eating fewer donuts might be a good idea for other reasons, but might not help you get into this form.
So how does this kind of thing get through automated tests? A couple of hypotheses: even though the form field isn’t possible for a human to access, the field has been rendered, so an automated test looking for that field will find it and fill it out. Alternatively, perhaps this phone’s screen is uniquely sized to cause this problem to happen, and the automated tests were only working against one or two screen sizes.
In a mobile-first world, you’d be surprised (or maybe you wouldn’t!) how often companies neglect testing on actual mobile devices.
This concludes the first edition of “bugs people find.” Stay tuned to the blog and we’ll keep you informed with fascinating (okay, if you’re into that) tales of bugs our testers uncover.