Exploratory Testing and Pizza

What do exploratory testing and pizza have in common?

Read on to find out; but first, let's talk a bit about what exactly exploratory testing entails.

Exploratory testing is an approach to software testing that emphasizes the engagement of the tester, the unscriptedness of testing, and the tester’s freedom to go down paths they think are important (not just one prescribed). Empowered to choose what they focus on and how they evaluate the software, these testers can pursue tests in areas that have potential for problems or that they think (based on issues in other areas) should be checked out.

Test design, exact testing steps, and expected outcomes for exploratory testing are not determined in advance (that said, guided exploratory testing does provide more room for specificity, as mentioned below). Thus, while scripted testers follow a set of test cases, and the paths in the software they are checking have been considered and laid out for them in advance, exploratory testers get their results without following a predetermined route.

This approach to testing values testers’ experience, intelligence, creativity, learning, and even intuition by locating test design, test execution, and test insights organically. Exploratory testers are freed from the encumbrance of documenting their steps in advance, which gives them the chance to incorporate what they learn, as they learn it, directly into the next test.

Of course, this doesn’t mean the testers are doing whatever they like; exploratory testing isn’t random clicking or pushing of buttons. Yet, by removing strictures of scripted testing, exploratory testing allows for a more holistic approach to test design, execution, and analysis.

Reduced test preparation time — Since this testing methodology requires less documentation, the tester is able to start experimenting sooner.

Simultaneous learning — Skilled testers can utilize their personal skills, experience, and knowledge to the testing process and learn about system behavior while testing. This means they can design, execute, and even improve testing scenarios in real-time, seeing how scenarios or steps connect with an agility that can’t be replicated by automated testing.

Dynamic feedback — With testers freed from the traditional test-case based approach, they are able to focus their efforts on key areas or problem areas as needed. This can give developers much needed input on product quality or possible issues on an ongoing and more immediate basis.

Finds critical bugs — Scripted tests are limited to some degree by confirmation bias. The developers will design cases based on what they think might go wrong. On the other hand, exploratory testers think differently and explore more freely to discover difficult-to-detect defects.

In a response to a question on Quora, Pradeep Soundararajan, author of Tester Tested!, takes a creative look at exploratory testing by likening it to cooking pizza:

“You can cook pizza by following instructions or trying it out on your own. When you try it out on your own, you learn a lot more than merely following instructions. You don't get pizza a couple of times but once you get it, you would have exhausted many mistakes. The next time you have to cook any food, you will be aware of the kind of mistakes you made with pizza and would then ask a bunch of questions to avoid doing those mistakes. You may ask for the instructions but you won't treat the instructions the same way you would have treated had you not done your own mistakes. You have prepared yourselves to cook any food in this exploratory cooking process.

However, you get a pizza faster when you follow instructions given to you. The goal is to not produce one pizza but thousands and if you need to keep following the instructions every time, the first pizza is faster but to produce thousands is slower. Accidentally if the instructions are wrong, you perfect the imperfect pizza. The next time someone asks you to cook pasta instead of pizza, you are asking instructions, follow it and pray God that the instruction given to you is perfect.

There are competitions like Master Chef and what we observe is - the participants need to cook food and there are some standard procedures to follow to cook the same food. However, some people do it more awesome than others. Why? Their experience of doing mistakes with the dish, their skill and passion for doing it better and their skill to make the right decisions at the right time helps them win.

Be a master chef. If you want to be one. Be one from today because nobody becomes somebody all of a sudden.”

(Yesterday, I shared another great analogy for exploratory testing that was formulated by Alex Schladebeck.)

Confirming that the app or software runs smoothly inside your office doesn’t guarantee it will work all of the time for everyone. Your users may be using your product at home, at work, on the road, with spotty Internet, in rural or urban areas, on mobile, laptop, desktop, or even IoT devices. That’s why exploratory testing techniques focus on identifying:

Product purpose — What is the fundamental thing the product is meant to do?

Primary functions — What are all of the primary functions the product does?

Areas of potential instability — Which functions are most likely to fail, crash, violate stability standards, be unreliable, or have negative side effects?

The actual testing of each function and recording problems then follows. The aim is not to spend a lot of time examining every possible pathway until defects are detected. A diligent tester will take a deeper dive in identifying issues otherwise difficult to find. Based on their familiarity with all that can go wrong during the course of software development, these testers focus on how users might interrupt, corrupt, crash, bend, or break the system.

Nevertheless, exploratory testing is anything but making wild and uneducated guesses. As James Bach wrote in an often-cited paper on exploratory testing, the prime directive of this methodology is to “be thoughtful and methodical.”

Once you’ve identified the best technique for you, you’ll need to find the best testers.

Inquiring — Exploratory testing doesn’t pre-determine what’s to be tested or how, so great exploratory testers are inquisitive and interested. They want to find out everything. They want to know what’s behind this door (or button) or and what’s hiding in this box (or feature). They need to be driven to learn more since they’re on their own when it comes to checking the different parts of the website or application and investigating its nooks and crannies.

Flexible — Great exploratory testers are competent test designers who can roll with the punches: if an investigation of a specific part of the software turns up an error, they need to be able to improvise new tests that look for related problems and issues. An exploratory tester who is good at coming up with impromptu additional tests can also determine the scope of a bug, or at least circumscribe the impact of it, which makes the task of fixing it easier for the developer.

Observant — Inquisitiveness isn’t enough on its own. Exploratory testers should not only be curious, but also able to notice any aberrations and document them. An excellent exploratory test requires the testers to observe settings, environmental factors, and internal mindsets as well.

Holistic — Exploratory testing needs to consider and record anything that may influence the testing. This could a range of factors such as: wifi versus cellular, signal strength, battery level, or even ambient noise. The great exploratory tester embraces the need for additional investigation and fully considers all paths that should be investigated instead of settling for one scripted solution.

Excellent communicator — Someone can be intelligent and inquisitive and awesome at finding defects, but it does little good if they can’t communicate their test findings to the developers. Without a separate test designer sitting down with the product manager to pre-determine individual testing steps and expected outcomes, testers are responsible for explaining the idea behind a given test, the circumstances surrounding it, and then the outcomes that result.

The best exploratory testers know when to use words, when to use screenshots, and when to take videos. They take into account the point of view of the people who will be using their work: developers, QA professionals, and product managers and offer the key information those people need to do their work.

Focused — Great exploratory testers also know their aim is not to be comprehensive, covering every scenario. They are to find defects and issues that wouldn’t be found through typical testing. At the same time, while they are looking at edge cases and parts of the system away from the normal usage pattern, they are focusing on the areas that are less likely to be well tested otherwise.

Exploratory testing can be an integral tool to virtually any test suite if properly implemented and should not be shirked in place of robust automation practices. Both are necessary for different testing scenarios, and both can yield incredible results.

That said, since exploratory testing unleashes the full potential of each and every tester, it makes sense to have a large crowd of experienced testers to choose from. That's where crowdtesting comes into play.

For us at test IO, our experienced and engaged testers can ensure the kind of coverage required for high-quality software and let you benefit from the wisdom of the crowd. Your software gets the benefit of exploratory testing and the unfettered examination of an experienced software tester, counterbalanced by the security of having a crowd to find all the important bugs. We can quickly put dozens or hundreds of testers to work testing your software, app, or website based on the exploratory parameters you outline.

Enjoy real-word tests from a diverse group of global professional testers using the varied devices, operating systems, and browsers you want to test. Learn more!