Testing in Production: From Punchline to Process

November 30, 2016

The customers who test in production with test IO fall into three categories:

  • The business moves so fast that the cost of being late exceeds the cost of being wrong -- such as putting up new offer pages on a travel site.
  • The team we’re working with is responsible for deploying the code it’s been given, whether or not they’ve verified that it works themselves. Better to deploy and test immediately than to wait and find out from actual customers that something doesn't work.
  • Technology leaders feel confident enough, due to a combination of crowdtesting on a feature branch and continuous delivery hygiene, that they see no need to slow down the release train to perform a final test on a staging server.

If you fall into any of these categories, here are a couple of things to think about if you want to test in production:

  • test IO’s API lets you invoke a test automatically. Suppose you do a production push at some point when your customers are least active on the site. You can call our API and start a test using test cases, exploratory tests, or a combination of the two. We can tune the tests to return the kinds of issues you care about most when code is in production.
  • If you can push your new code behind a feature flag, our testers can exercise the code before you turn it on for your customers. So if you’re responsible for the customer experience but not able to make code changes, run-time feature flags combined with crowdtesting let you make sure code is safe for everyone before you turn it on.
  • Remember that you can ask our testers follow-up questions when they’ve filed bugs, so if you’re monitoring and seeing errors from your monitoring tool in production, we can help you with debugging.

Just as a map of the territory is not the territory, a staging environment is not a production environment. Crowdtesting behind a feature flag is the closest you can come to giving your software to customers in real-world conditions without risking a dime of revenue or brand equity.

So maybe it’s not so reckless after all.

Let us know if you want to learn more.

Read More

March 12, 2020
Tester Spotlight - Alex Che

We’re able to do what we do because of our incredible and diverse community of testers. Meet Alex, a Policeman now QA Engineer who gained the real-world QA experience he needed to change careers testing with test IO.

March 10, 2020
A Benefit of Crowdtesting - Time Compression

When budgeting for crowdtesting in the coming years, it's important to know all the value it adds. Here a little more on one of those values, time compression.

February 27, 2020
A test IO Bug's Life

You click submit on a new crowdtest and go home. When you wake, detailed bug reports are waiting in your Jira. But how did they get from the tester to you?

Ship Faster, Sleep Better

Get a Demo