Testing in Production: From Punchline to Process

The customers who test in production with test IO fall into three categories:

• The business moves so fast that the cost of being late exceeds the cost of being wrong -- such as putting up new offer pages on a travel site.
• The team we’re working with is responsible for deploying the code it’s been given, whether or not they’ve verified that it works themselves. Better to deploy and test immediately than to wait and find out from actual customers that something doesn't work.
• Technology leaders feel confident enough, due to a combination of crowdtesting on a feature branch and continuous delivery hygiene, that they see no need to slow down the release train to perform a final test on a staging server.

If you fall into any of these categories, here are a couple of things to think about if you want to test in production:

• test IO’s API lets you invoke a test automatically. Suppose you do a production push at some point when your customers are least active on the site. You can call our API and start a test using test cases, exploratory tests, or a combination of the two. We can tune the tests to return the kinds of issues you care about most when code is in production.
• If you can push your new code behind a feature flag, our testers can exercise the code before you turn it on for your customers. So if you’re responsible for the customer experience but not able to make code changes, run-time feature flags combined with crowdtesting let you make sure code is safe for everyone before you turn it on.
• Remember that you can ask our testers follow-up questions when they’ve filed bugs, so if you’re monitoring and seeing errors from your monitoring tool in production, we can help you with debugging.

Just as a map of the territory is not the territory, a staging environment is not a production environment. Crowdtesting behind a feature flag is the closest you can come to giving your software to customers in real-world conditions without risking a dime of revenue or brand equity.

So maybe it’s not so reckless after all.

Let us know if you want to learn more.