DevOps Best Practices: Integrating QA and DevOps
The emergence of DevOps as a practice over the past five years has helped make software quality every engineer’s responsibility, and by and large this is a good thing. But if quality is everyone’s responsibility, that also means it’s no one’s responsibility — and customers can suffer the consequences. While many organizations no longer have explicit QA departments or “testing phases,” the practice of quality assurance remains the best way to answer the question, “Should we ship this to customers?”In this article, we’ll explore how QA can train your team and strengthen your project, coaching you toward that dramatic moment when you hear the whistle blow, and the big game, known as shipping to your customers, has begun!
The disconnect between QA and DevOps often occurs due to misconceptions about the role of quality assurance in the modern software development life cycle. Even as continuous integration servers and continuous delivery practices check your code for errors, they don’t ensure its quality when you define quality from your customer’s perspective rather than your code’s: “quality is fitness for your customer’s purpose.”
Think of it this way: quality assurance isn’t just about finding bugs; modern QA is a fitness regimen for both your application and your organization.
Continuous delivery is undoubtedly a powerful tool when used properly. Releasing new, fully-functional builds at the drop of a hat is an alluring prospect, but it can be a dangerous siren song if poorly implemented. With fully-automated workflows, all builds that pass automated testing are pushed to staging (or even production), but how do you decide whether customers should actually see it? The delivery of a final product to customers is a business decision, not a technical one. Such business decisions are so critical that they are best made with multiple inputs, providing a more comprehensive view of the risks.
Even sophisticated organizations like Google experience issues with continuous delivery practices, due to a lack of human-oriented quality control practices. In one particular case, the Google Consumer Surveys team pushed a build to production that contained an obvious visual bug: a dancing purple pony. The cause of this major fumble? An incorrect CSS class name. The team at Google had implemented the right series of checks and balances to verify the integrity of the build and ensure nothing untoward was pushed to production:
Everything came back green. All tests were successful, staging looked as expected, and even their QA checklist failed to notice the issue. In spite of all these checks, along with the dozens of metrics that failed to report any issues, none of their systems could identify this dancing purple pony sitting right there on the page. As Brett Slatkin, engineering lead on the project, puts it, “I don’t know anything about how my customer perceives the app.” [@13:15]
In the old days of waterfall development and separate QA teams, Slatkin or his product manager had an independent source of information about his product’s fitness for customers. Today he might not. In Google’s case this is a trivial example for a relatively marginal product. But if (like most companies) your risks are higher and your level of automation lower, you need an independent source of truth about your customers will perceive the app.
As we’ve seen, even the most diligent organizations can confuse good test results with actual fitness. If teams assume that automated tests, code review, and staging deployments are all it takes to be fully prepared to climb that mountain of production release, it shouldn’t come as much of a surprise if an unexpected fall occurs, plummeting the project back toward the bottom.
A handful of op-ed pieces and blog posts have appeared lately, suggesting that the need for QA is declining with the rise of DevOps practices. The core of this contention is that automated methodologies are sufficiently advanced that human intervention is largely unnecessary in the release process. Speed and efficiency of computerized solutions wins out over traditional QA methods, or so the theory goes.
Two counterpoints are critical to consider here:
Ultimately, quality assurance should emphasize the customer experience over all else, by ensuring that software quality is at peak performance.
But how can you get a regular sense of your software’s fitness for the Big Game if you’re operating with the speed and efficiency of a DevOps team?
Historically, quality assurance has been relegated to the back of the pack, as one of the very last picks in the disciplined pickup game that is the development life cycle. Waterfall methods are notorious for pushing testing and other quality control practices toward the end of the process, which often leads to a mad sprint to fix bugs and reach the finish line of release, before the looming deadline.
Modern agile methodologies, which are rough and tumble by comparison to waterfall styles, are typically more test-oriented, ideally centering testing among the engineering team. Yet, even with more diligent developer-driven testing practices in place, agile DevOps risks relegating customers-focused QA to an afterthought. Frequently exercising your product with real-world people and customers is a smart business decision, ensuring the system doesn’t become out of shape. An ideal world of software development combines automation with human judgment, informing better business decisions, including when your software is totally fit and ready to ship to customers.
Among DevOps-savvy organizations, we see three different patterns for integrating human-driven QA and DevOps:
At test IO, we’re a crowdtesting company, so we see companies that have integrated crowdtesting into their release process. It’s certainly possible to follow these practices solely with an in-house team, and better still to do it with an in-house team that’s supported by crowdtesters. One key value of crowdtesters in this equation is speed. The on-demand model at test IO means that we can bring the appropriate resources to bear on a project at speeds that are difficult for most companies to achieve purely with internal resources.
The integration of a QA process into DevOps helps you manage risk and ensure that your application is both stable and robust. Bugs will be found earlier and more often. But most importantly, it makes sure your application is always fit for game time. Just as a great athlete is not defined solely by VO2 Max or pulse at rest, your app isn’t defined by DevOps test scorecard alone. Human insight provides that extra level of fitness through continuous improvement, making sure you’re training for the real goal: building great software for your customers.