Continuous Testing Manifesto

Continuous testing...

• combines automated testing and testing with people to ensure full test coverage in real life situations for your products
• makes the most of computers' precision and humans’ creativity to find bugs before they can cause damage
• turns testing with people from the major drag just before shipping into an essential part of development, just like automated testing
• can be used to solicit tester feedback to improve and shape the product from the beginning of and throughout the development process

While automated tests are awesome and an integral element of the agile software development process, they are only part of the story. The majority of software is written by people and used by people - and thus should also be tested by people.

Computers are precise and logical, which makes them perfect for automated testing - but even those tests are written by people and often can't cover all eventualities.

Humans may not be so precise, but they outdo computers in many other ways: creativity, context-awareness, outside-the-box thinking and – a clear advantage in this case – being somewhat unpredictable. Combining automated testing and human testing is what continuous testing is really all about: ensuring full, real-life test coverage for your products.

While automated testing has found its way into the processes of many development teams, testing with real people in the real world often remains a time-consuming, necessary evil just before the product launch.

Officially, testers are supposed to uncover bugs, but unofficially, everybody secretly hopes they will not come up with anything major that would delay the scheduled launch. The reality is, however, the earlier a bug is found, the less damage it can do and the fewer costs it give rise to.

Continuous testing uses this insight by integrating testing at the very beginning and keeping them involved throughout the development process. This prevents bugs from disrupting the development plan and keeps expenses down.

A continuous testing scenario implies real, working continuous integration. As soon as code is pushed to the CI server and the automated tests run green, it gets deployed to the staging server where testers can screen it for bugs.

Detected bugs are submitted to developers, who can then fix them right away, and ideally write the corresponding automated tests so the same bug doesn’t turn up again.

This turns human-driven testing  from a major impediment to shipping the product to just another part of development, like automated testing.

By implementing continuous testing, tester feedback is available early on to refine and improve the product, not just as the end when it might be too late. all the way through the development process.

While testers are looking for bugs, at the same time they can provide crucial feedback on usability and features, continuously complementing market research and public beta testing.