Testing in Production: From Punchline to Process

November 30, 2016

The customers who test in production with test IO fall into three categories:

  • The business moves so fast that the cost of being late exceeds the cost of being wrong -- such as putting up new offer pages on a travel site.
  • The team we’re working with is responsible for deploying the code it’s been given, whether or not they’ve verified that it works themselves. Better to deploy and test immediately than to wait and find out from actual customers that something doesn't work.
  • Technology leaders feel confident enough, due to a combination of crowdtesting on a feature branch and continuous delivery hygiene, that they see no need to slow down the release train to perform a final test on a staging server.

If you fall into any of these categories, here are a couple of things to think about if you want to test in production:

  • test IO’s API lets you invoke a test automatically. Suppose you do a production push at some point when your customers are least active on the site. You can call our API and start a test using test cases, exploratory tests, or a combination of the two. We can tune the tests to return the kinds of issues you care about most when code is in production.
  • If you can push your new code behind a feature flag, our testers can exercise the code before you turn it on for your customers. So if you’re responsible for the customer experience but not able to make code changes, run-time feature flags combined with crowdtesting let you make sure code is safe for everyone before you turn it on.
  • Remember that you can ask our testers follow-up questions when they’ve filed bugs, so if you’re monitoring and seeing errors from your monitoring tool in production, we can help you with debugging.

Just as a map of the territory is not the territory, a staging environment is not a production environment. Crowdtesting behind a feature flag is the closest you can come to giving your software to customers in real-world conditions without risking a dime of revenue or brand equity.

So maybe it’s not so reckless after all.

Let us know if you want to learn more.

Read More

June 13, 2019
The Important of Manual-Stage Testing

Incorporating manual testing into your development pipeline, without bottlenecks.

June 7, 2019
Regression Testing: Where Does Crowdtesting Fit In?

What is regression testing, and how does crowdtesting benefit it?

May 30, 2019
HUM Nutrition: A Test a Day Keeps the Bugs Away

test IO helps HUM Nutrition provide customers with life-changing products.

Ship Faster, Sleep Better

Get a Demo
testIO Logo Footer
test IO Inc.
1111 Broadway, Floor 3 
Oakland, CA 94607

test IO GmbH
Sonnenallee 223 a
12059 Berlin