Software QA Testing for Marketers: Check Your Conversion Rates!

When you’re a marketer, you dream of funnels. In marketing-speak, a funnel is the Darwinian process of selection that winnows the millions of potential customers you might have to the rather smaller number that you actually get. Suppose 1,000 people come to your site, 100 view your product, 10 click “buy” and 1 person makes a purchase. That means only 10 percent of the people with a demonstrated intent to purchase actually spent money -- as a marketer, that’s an abomination! So you do everything you can to optimize the purchase flow: the words you use, the number of clicks required, the speed.

But what if the problem is in your code itself?

This happens a lot. New code goes out. Conversion rates dip slightly. Maybe it’s not enough to worry right away, or maybe it’s just because it’s Tuesday, or seasonality, but after a week or two it’s serious.

As a marketer, you have very granular tools to see who’s coming into your funnel and where they exit. And you can segment your audience by demographics, geography, operating system, browser, and screen size.

There you find something interesting -- your conversion rate has dropped among on one particular device. And when you investigate, you find that on this device, customers need to scroll to see the buy button, or a pop-up blocker on a specific browser prevents them from making a purchase. With enough users, this means real revenue.

Generally, if you’re seeing very different conversion rates from different devices or browsers, or if you see a sudden inexplicable conversion rate decrease, you might have a bug. In the mobile world, it used to be true that Android conversion rates were much lower than iOS conversion rates, but that gap has narrowed to around 5 percentage points. If you’re seeing a larger gap than that, you may have bugs in your Android experience that are preventing your customers from spending money.

As the marketer, you can kick yourself that such a thing could happen to you. You can go talk to your development team and receive well-meaning assurances that they’ll catch it next time. Or you can make sure it doesn’t happen by testing your software in the real world, with real users, on real devices.

At test IO we work with many different teams: sometimes it’s the QA team itself, other times it’s a product manager, or a DevOps team, or an engineering team. But for some of the most prestigious brands we work with, we’re working with marketers.

In some ways this is not surprising, since marketing has become much more quantitative and rigorously “experimental” over the past few years. For savvy marketers with prestigious brands and significant revenue on the line, software testing is of a piece with the other kinds of testing that they do as marketers, not a concept reserved for technologists.