DevOps and Software Testing

DevOps is a software development and delivery process which considers the production of software from end to end, from concept to customer satisfaction. While Agile is about the delivery of a quality software build, DevOps expands the scope to include deployment to customers and feedback from customers. DevOps includes all the stakeholders involved in developing the product including product managers, developers, quality assurance, operations, system administrators, management, security and support. It’s about the team understanding software isn’t complete until the user accepts the software.

DevOps is not a technology, a standard or a process. It is a movement. To successfully implement DevOps in an organization, a change in culture and mindset is required. DevOps is largely about implementing organizational change to accomplish its mission. This means removing the barriers that typically exist between two siloed teams: Software Development and Operations. In a DevOps world, the two teams work together to maximize the productivity of developers and the reliability of operations.

In a fully implemented DevOps culture, team members view the entire infrastructure and development lifecycle as part of their responsibilities. Their focus is satisfying the customer rather than having a more limited perspective like keeping a server running or pushing the latest build. With this focus on the customer, DevOps emphasizes collaboration and frequent communication among members of the team.

In a DevOps environment, software is built, tested, released and deployed more rapidly, frequently and reliably than with other approaches. The collaboration within the team may result in shared techniques. For example, the Operations team may use Scrum, Kanban or other Agile methods. Deployment scripts and server configuration scripts may be placed under source control.

Migrating to a DevOps culture requires an investment of time and effort. The effort is rewarded in several ways. With DevOps, reliable software is released faster and more frequently. Time to market is accelerated, improving the product’s competitive position. The time from the creation of a requirement or user story to the time the feature is deployed to a customer is shortened, improving the customer’s experience. If a showstopper bug is discovered, the time to recover is diminished.

DevOps is not a standard and there is no requirement to do everything described here. Every software development effort has its own constraints which may affect a team’s approach to DevOps. While it’s not a simple task to fully implement DevOps at once, an incremental approach can be simpler to manage and still yield significant rewards. These techniques are typically implemented in a DevOps effort.

If you are beginning a new software project, you may want to consider implementing a microservices architecture. A microservices architecture is a suite of small, independent modular services that communicate with each other, typically via HTTP/REST. This approach can improve scalability and reliability and greatly simplify the QA effort. Microservices allow teams to work simultaneously on separate services, improving throughput. The model of isolating the code into separate services which communicate via a REST interface reduces code coordination issues and simplifies the task of writing tools that test the services.

If you have an existing product, the transition to a microservices architecture may not be practical. The rest of the DevOps techniques described here are still viable for your project and can result in the major gains associated with DevOps in general.

Rapid delivery is a fundamental part of DevOps as it is in Agile Development. Software is delivered in small iterations. Because the changes are smaller, it is easier to isolate and resolve newly introduced bugs. Smaller iterations are fundamentally less risky. The concept of minimizing risk and identifying problems early pervades DevOps.

Rapid delivery is not merely confined to frequent, small builds in DevOps. A major goal of DevOps is to automate the software delivery process as much as possible. Builds, integration, testing, deployment and logging can all be automated- greatly improving reliability of the product while improving development velocity.

DevOps automation and risk management can be applied to the build process. Every time a developer checks in code, it is automatically deployed to a build server. From there, an automated build is executed including automated testing. This approach assures early detection of bugs and minimizes code integration issues. It’s faster and less risky than manual approaches, but it requires a commitment to automated testing.

While continuous integration is about developers checking their code smoothly into the main branch, continuous delivery is about automating the release process so the application can be automatically deployed at any time. In DevOps, when the code passes continuous integration testing it should be automatically deployed to a test environment separate from the dev environment. The test environment should mimic the production environment and therefore be refreshed before each deployment to assure a clean target. The refresh of the test environment should also be automated. Once the build passes QA, it is ideally deployed to a staging environment which is identical to the production environment (as opposed to the test environment which is similar but has all the QA tools installed).

A final test on the staging environment assures there are no deployment issues, then the application is deployed to production. This migration of the build from server to server, with the server provisioned each time in advance of the move would typically be the responsibility of Operations. In the DevOps world, Operations and Software Development would plan this process together and implement Infrastructure as Code (IAC), greatly improving the likelihood of an efficient, reliable and smooth automated delivery process.

Software testing in the DevOps world requires an automated test process that provides feedback at every checkpoint. Integration testing is needed on the build server. Functional testing, regression testing and more are needed on the test server. Deployment testing is needed on the staging server. If the build passes, it advances to the next step. If it fails, it’s time to stop and fix the issue.

Of course, not every development team is able to do every type of testing. Resource constraints frequently limit a team’s ability to implement DevOps. Test automation takes time and it’s rarely possible to automate every conceivable test. For this reason it’s practical to define an automation strategy. Automated test development should be prioritized to critical areas like key functionality or areas of complexity. It may be productive to chase out early bugs in new code before automating, as the bugs may hinder automated test development. Regardless of your automation strategy, it’s wise to keep your test scripts under version control.

The goal of DevOps is to satisfy the customer. Throwing your finely polished application over the wall and waiting for support tickets to come in is not the best way to achieve this goal. The DevOps team can do quite a bit better with continuous monitoring. With continuous monitoring, you monitor the health of the application as it runs in production, checking performance, errors, resource consumption and other conditions. The DevOps team may use monitoring tools or build self-monitoring or analytics capability into the application itself. With continuous monitoring, the team can spot issues before the customer and take corrective action.

The complexity of software testing in the DevOps world of fast-paced deployment demands flexible thinking about deployment of test resources. Risk reduction is paramount and crowdtesting can be a critical component in that effort. Testing may include:

- Integration testing
- Functional testing
- API testing
- Exploratory testing
- Regression testing
- Compatibility testing across platforms
- Security testing
- Acceptance testing
- Deployment testing

To maximize test agility, a solid strategy is to deploy testers on demand for specific types of tests or specific functionality. For example, testing your new functionality on a wide range of mobile devices is needed, but it can be difficult to assemble testers with the desired devices and configurations quickly. Crowdtesting by Test IO solves this problem by giving you the ability to quickly assemble the testers you need with the right devices and configurations to rapidly test, provide feedback and keep you on pace. Similarly you might perform exploratory testing via crowdtesting to flush out defects in high risk portions of the code.

DevOps represents a major revolution in the way software is delivered. It offers faster delivery of product and an improved customer experience. It reduces the risk of deploying deficient software, and that makes everyone happy: senior management, software development, sales & marketing, operations and most importantly, the customer.