How to Scale Manual Mobile App Testing

The mobile industry is showing no signs of slowing down. Apple, Google, Samsung, and many others are rolling out new Android and iOS devices faster than ever. Those top 3 alone released ~23 unique mobile devices in 2019. The total number of users worldwide who use mobile devices instead of desktops reflects this. Statista states that 52.31% of people use their mobile devices instead of their desktops.

Companies have responded by focusing on developing pristine apps to reach their user base where they are most, their phones. However, the ever-increasing device and OS pool have not made shifting to mobile easy. Now Quality Assurance teams must always have these two questions on their minds: 

1. Will the app perform seamlessly across all the latest Android and iOS devices? 
2. Will the app be compatible with older versions of operating systems running on Android and iOS? 

This article will address how you can assuage those two concerns. How manually testing mobile applications can play a crucial role in your development process. Finally, how one can efficiently carry out online manual mobile testing, cost-effectively, on-demand, and at scale.  

It's always an excellent day for QA when an app is compatible with the latest iOS and Android versions. However, it is just the beginning. Many users use older devices, and many more use outdated operating systems. In an independent study done by David Smith, David found that overall only 9% of his users were using the latest Apple Devices, and 92.5% were running the latest operating system. That means an overwhelming percentage of his users were not on the latest devices and os combination.

While an isolated study, these numbers are an average representation of our population. They show that if you choose not to optimize your app for earlier versions of Android and iOS, it may lead to significant losses in revenue, users, and more. All the more reason for extensive mobile application testing.

So much focus has been given to automated testing and testing automation tools, that to some people, manual testing might sound redundant. At test IO, we disagree wholeheartedly. A machine cannot judge the usability, accessibility, and sometimes even functionality of an app. Humans bring experience, intelligence, creativity, learning, and even intuition to the table. If you'd like to read more about why that makes them exceptional testers, check out this post on exploratory testing.

Now that you understand why we need to test manually let's discuss three methods / mobile testing tools that QA teams can use to perform manual mobile app testing on different Android and iOS devices: 

Crowdtesting is the most flexible way to get manual mobile app testing done, and its the only way to scale your testing without significantly changing your team. With a provider like test IO, you get access to over 30,000 testers around the world, waiting to test your mobile app on their personal devices. The process is simple, connect the test IO platform to your software’s staging or production environment, set your test cycle requirements(which devices you want to be tested), and unleash the crowd. In as little as an hour, our testers return high-quality bug reports, usability feedback, and even positive validations of user stories. That’s it!

At test IO, these crowdtesters don't need specific scripts or test cases; teams usually ask them to evaluate sections or features of the application, using their experience and best judgment to unearth and identify problems. The capabilities of the crowd can change the testing framework at a company entirely. Not only does it give access to real users who can best replicate your user's experience, but crowdtesting can also compress months' worth of testing into one night and radically increase the test coverage of your team. If you want to learn more about how crowdtesting can become one of your best software testing tools please check out this article.

A device farm is one of the easiest ways to get started manual testing. This kind of platform will allow you to test your mobile apps through the use of a real device cloud that hosts various Android and iOS mobile devices. Your QAs can use platforms like this to access nearly any device they could want to perform real device testing without having to build a device bank in house.

While helpful, this testing solution does not scale easily. If you want to test across multiple devices, which you should, your QA's will still have to perform the same test scripts over and over again. As your product grows and devices change, this will eventually become unsustainable.

QAs need to identify all the devices their app must be tested on and download emulators for each device. As we've made clear, that list of devices is extensive and always changing. Downloading and testing on each android emulator or ios simulator can become cumbersome and inefficient. Plus, like a device farm, your QA team is still tasked with doing all the testing. Many teams can't afford to adopt such an approach with release times speeding up and any little bug making or breaking a release.

Don't get me wrong; emulators are very useful, especially in the initial phases of development. However, they do not accurately approximate real user conditions. So if you only test on Emulators, you run the risk of missing the bug that ruins your release.

The smartphone industry is going to continue to grow. To keep up, you have options. Use a device farm that's easy to set up, but time-consuming as your team tries to run tests on as many devices as possible. Painstakingly download and manage emulators and simulators for every possible device and os combination, but be left with testing that doesn't accurately simulate your user experience. Partner with a crowdtesting service like test IO which provides instant access to 30,00+ testers on real devices that can execute what would be weeks of manual tests in just a few hours. As the number of smartphones continues to grow and the need to scale increases, you know which one I would choose.