Black Box Testing

There are two primary — complementary — types of Quality Assurance testing. White box testing, a widespread style of testing, sees the programmer writing tests for his or her own code. Unit testing, integration testing, regression testing, and coverage tests all can fall into this category. White box testers have full access to the source code and understand how the software has been created.

Black box testing, on the other hand, is done by testers who don’t know anything about how the software has been put together. Quite simply, instead of looking into a translucent (white) box, they cannot see inside the black box they are presented with.

Their lack of knowledge of the internal structure doesn’t mean that black box testers are testing completely blind. The client will typically provide a brief outlining what the software is supposed to do (as outlined below). In the case of functional testing and test cases, the brief would cover what specific functions or usages the testers should try.

With this testing method it is possible to find incorrect or missing functions, as well as initialization and termination, behavior or performance, data structure or external database access, and interface errors. Tested inputs include data, clicks, scrolls, touch, voice, feeds or any other means a user might engage with the software.

When you’re a white box tester, no matter how hard you try to do otherwise, you test the software based on your preexisting knowledge of the development and design of the product, and with the same assumptions made by the code itself.

Since unbiased, motivated, and diverse human testers are best, white box testing alone isn’t enough. With independent testing, it is much easier to identify contradictions or vague functional specifications as the tester is performing tasks from a user’s point of view rather than the designers or developers’ perspective.

Instead of “testing to the code,” black box testers can discover a completely different set of bugs and issues than the original developers since they are coming to the software without any preconceptions. In fact, black box testers don’t even need to know the programming language used to develop the software since they are testing cases without access to the inner structure. The goal is for them to interact with your software as a user might — and there are few users out there that know their Ruby from their Perl. Actual user input comes into play later on with beta and acceptance testing.

Another advantage is that the test cases can be designed once specifications are complete, which can help make the testing process more efficient. This also provides input sooner in the process, which can lower costs since it takes less time to fix defects earlier in the development cycle. This makes it particularly useful in testing large systems.

Black box testing is also more open-ended, since testers aren’t privy to the internal details of the system or what developers expect users to do.

The testing in this method can be functional or non-functional, though this is more often functional testing with the designer selecting the valid and invalid inputs as well as the correct output. Testing all possible combinations of input values for the software under test (SUT) is exhaustive and impractical most of the time. Thus, black box testing typically analyzes input/output, observable behavior, and heuristics via several complementary techniques.

Equivalence partitioning — Valid at all levels of testing, equivalence partitioning divides valid and invalid inputs into representative classes to derive test cases. This testing assumes the program will perform analogously for equivalent inputs. If a fault is detected the assumption is that the same fault will be detected for others in that class.

Boundary value analysis — Also valid at all testing levels, boundary value analysis tests the boundaries between equivalence partitions. This testing will look at maximum, minimum, just inside, and just outside boundaries as well as typical and error values.

Cause-effect graphing — This analyzes combinations of input conditions to identify cause and effect, or test combinations of inputs that produce different results. This black box testing technique is typically used to test the system’s external behavior. This approach is similar to decision table testing, but as the name suggests this one uses graphs (rather than tables) to visualize the inputs, outputs and flow.

Decision table testing — Used in complex scenarios to examine combinations of inputs resulting in different outputs, decision tables can offer testers a systematic view of various possibilities. This approach to testing often states conditions and actions in Boolean (true or false) structure to create combinations that otherwise might not have been anticipated. Conditions represent the input while actions are the events the actions should trigger.

State transition testing — A system is seldom in one finite state, so this analysis allows the tester to see what happens when inputs trigger changes or when the software is transitioning between states.

Random & Stress testing — Running rare scenarios as well as typical ones helps increase the chance of triggering a system fault. Testers also design test cases to behave in extreme way in order to identify faults. This might involve emulating a denial of service attack, a huge amount of information being sent as quickly as possible, or a massive number of web users on a given application simultaneously. Ultimately, this type of testing aims to anticipate all of the crazy things a user might do to break the system.

Error guessing — Experienced testers can use their intuition and background in black box testing to “guess” where there may be errors. Knowing possible errors or error-prone situations, the tester may aim to identify high risk code sections or areas that could bear to be more thoroughly tested. The disadvantage, and why this approach is not implemented on its own, is that the tester needs to be very experienced (or clairvoyant) to write effective test cases.

The black box method can be employed in Integration, System, and Acceptance testing, but as the testing gains complexity it is likely that more than one different testing technique will be necessary. Fortunately, these testing techniques can build upon one another.

Let’s consider some of these black box testing techniques in action. Imagine, if you will, a tester interacting with a new gardening game app.

An early interaction with the app might test the registration process. For example, to test an input text box in which a user is required to enter their birthdate, the tester might begin with equivalence partitioning.

Instead of testing every imaginable number, letter, special character and combination of those three, they can identify groups for which to design test cases. For instance, they would generate a test case to validate that numeric input is functional and other test cases to confirm that inputs of alphabet, alpha numeric, and special characters are deemed errors.

The tester might next move on to boundary value analysis. The birthdate is meant to be entered as DD-MM-YYYY. Yet the tester will want to identify what the system does in response to several other values such as:

A single digit (minimum boundary)
Six digits (inside boundary)
Eight digits (maximum)
Nine digits (just outside boundary)

For the same app, testers might also employ state transition testing. Perhaps the game allows users to collect seeds to trade in for new plants, garden implements, or improved irrigation systems.

The tester would need to examine the software’s functionality when the user trades in his or her seeds. Their user status may change from Keeper of the Dead Plants to Green Thumb Gardener, which could trigger new actions. Plus, the user needs to be able to look in their inventory (or tool shed) for the new item. Also, the tester might ensure that when the virtual gardener tries to buy something else, the app knows that they can no longer afford it (having just depleted their store of seeds).

Black box testing has many advantages. Yet, in preparing to work with black box testers it pays to be aware also of its challenges.

Considering the testers are approaching the test case design without clear functional specifications, it can be more difficult to design test cases. Also, time it limited in most testing environments, yet the tester must develop specific testing inputs and identify all possible inputs without an insider’s knowledge. This can lead to testers repeating tests that have already been performed by the programmer in white box testing. Finally, the tester may wander down an unidentified path during testing, which may or may not prove advantageous.

So, what can you do to best prepare for black box testing that detects the maximum number of defects with the minimum number of test cases executed?

A detailed document or brief giving testers might include some or all of the following:

An overall testing schedule
Clear documentation of test plan
Possible inputs and expected test case results
Carefully selected test cases which avoid duplication with prior programmer testing and reduce the overall number of tests
Prioritization of test cases
Task responsibility
Any risks requiring contingency planning

Did you know there are actually several colors of testing? They include:

Yellow box testing — checks warning used by apps
Red box testing — validates error messages
Green box testing — release testing to determine if the system is environment friendly and has no social implications
Grey box testing — combines white and black testing in a kind of maintenance test
Clear box/glass box/transparent box testing — all another way of phrasing white box testing
Closed box testing — another way of saying black box testing

No matter what you call it, testing should be iterative as it is an essential part of the development process.

Effective testing will consider both the successful path a customer might take as well as a path that might lead to software failure in the users’ eyes. Black box testing is founded on the idea that the best way to explore both options is to get unbiased, heterogeneous testing feedback.

At test IO, we use on-demand and easy to set up crowdtests to provide the real world feedback from real people that black box testing should offer.

test IO makes it easy to integrate black box testing cycles into development workflow by:

1) helping to clarify your testing objectives

2) managing the testing process

3) finding unbiased, motivated, professional crowdtesters — when you need them (even overnight)

Crowdtesting and test IO remove the barriers to efficient, rapid, and flexible black box testing. Get crowdtesting today!