The Downfall of Manual Testing

April 18, 2019
John Kensinger
John Kensinger

On his blog, Joe Colantonio, posits there are a few reasons as to why manual testing may in fact be "problematic" and entail a variety of "pitfalls." Although Joe presents a fantastic, in-depth introduction to automation testing, we would like to qualify his (arguably anachronistic) statement on manual testing.

He states that manual testing:

1. Uses a lot of resources

2. Is time consuming

3. Sometimes lacks proper coverage

4. Is often of a repetitive nature, and testers may become bored and make mistakes while testing

While certainly holding true in the past, his sum up of why manual testing may fall short can be addressed by the technology we have at our fingertips today.


Our response to the above four points:

1. Uses a lot of resources

Traditional manual testing can require a lot of resources when constrained by having to hire in-house to perform this testing. However, if you can share the personnel burden, or negate it all together, the immediate resource cost diminishes drastically.

For example, one of our customers quoted our services at "8 full-time hires but at 1/10 the cost." Given the average cost of hiring a full-time QA professional in the Bay Area, that's a nearly $1 million per-year service that we provide at 1/10 of that cost.

2. Is time consuming

It definitely can be... When you’re relying on a finite number of in-house team members to perform the manual checks. What if you extend your in-house team? For every additional team member, the total amount of time spent testing before satisfactory results may diminish, but at an additional resource cost. Now, what if you could maintain the speed of doing so without this cost? With crowdtesting, you’re able to reap the rewards of an extended team without the extended cost.

In 2018 alone, over 180,000 hours were spent testing for our customers, equating too over 22,000 workdays of testing! This means that on average, our customers tested for more than 900 hours, or 1/3 the yearly workload of a full-time hire, with our most active customers testing up to 8x that amount.

3. Sometimes lacks proper coverage

In terms of device coverage, there’s definitely a drawback to manual testing, as maintaining a device farm can be pricey and time consuming. You could use simulated environments to help deal with this issue, which is more cost effective, but you’d miss out on testing in a real environment more representative to what your users will experience. By leveraging the crowd, you can test your software on nearly unlimited device and browser combinations. The size of the crowd, for example, ensures that all device-browser-update matrixes are covered, while also ensuring accessibility to the newest devices out there.

Sure, having one or two individuals manually follow test cases or perform exploratory testing may take a lot longer than running a tool like Selenium; but, more often than not, with automation you’re limiting yourself to very specific user flows or functionalities (you may miss something a real user might experience in a live setting).

With crowdtesting, you can gain the insights of real people on real devices. And with us specifically, you can take advantage of guided exploratory testing to receive detailed bug reports, complete with screencasts under real-world conditions.

4. Is often of a repetitive nature, and testers may become bored and make mistakes while testing

We completely agree with this point; this is why having a more diverse and, when necessary, dynamic audience to perform your testing can help to keep testers’ perspectives fresh and unbiased. While mistakes are still possible, leveraging a higher volume of testers enables you to catch these mistakes, and anything else that might go unnoticed by a smaller team, much more quickly.


Overall, there’s nothing wrong or incorrect about Joe’s initial statements, but as software testing continues to evolve, there may be more alternatives than initially thought.

If you're thinking about giving crowdtesting a try, we recommend you read this short checklist first. Otherwise, if you want to learn more about what we do at test IO, click here.

Read More

May 7, 2019
The Software Development Life Cycle

What is the SDLC, and what does it look like today?

May 6, 2019
The One Million Stories Behind test IO's One Million Bugs

test IO's one-millionth bug!

May 2, 2019
BuzzFeed: Finding Bugs at Buzz-Speed

test IO helps BuzzFeed find bugs at an incredible speed.

Ship Faster, Sleep Better

Get a Demo
testIO Logo Footer
test IO Inc.
1111 Broadway, Floor 3 
Oakland, CA 94607
USA

test IO GmbH
Sonnenallee 223 a
12059 Berlin
Deutschland
twitterfacebooklinkedin