What is Mobile Testing?
Customers expect your app to run flawlessly on their phones, regardless of operating system, screen size, permissions, even signal strength on the train . Sure, you can do mobile testing with an emulator, but an emulator is a lab — and mobile phones live in gritty reality. Plus, if you have a serious bug, the app store submission process makes it slower and costlier to fix than the same problem would be on the web. Bottom line: mobile development demands a mobile testing strategy.
Mobile Testing with test IO
It’s expected there will be 270 billion mobile application downloads this year. Yet, it’s estimated that 80% of those apps will fail
. Mobile app testing positions your product launch to be one of the quality applications that not only survives but also succeeds.
Why mobile testing is needed
Competition is fierce. Consumer expectations are high. User patience is thin. The future of your hard work developing that application revolves around the UX. But a buggy app will not only lose users but can also damage brand reputation — quickly. Unhappy users may delete the app right away, more so, if it stays frozen for half a minute. They will share their horrible app experience with friends and family and post damning reviews in app stores that will undermine future product releases too.
Reasons for market failure of mobile applications
In a yet-to-be duplicated 2012 study from Apigee, American mobile app users
identified many things that would prompt them to post a bad review:Freezes
– 76% — In fact, “an unforgiving 18% admitted they would even delete an app immediately from their device if it froze for just 5 seconds.”Crashes
– 71%Slow responsiveness
– 59%Heavy battery usage
Mobile app testing helps you stay ahead of the curve. By identifying possible user interface annoyances, user experience stumbling blocks or functionality failings before launching a new product or the next iteration of your software, you can release a better product.
How mobile testing works
Just as there are myriad mobile devices available, there are also many different ways people can test mobile. These include device labs, emulators, cloud-based labs, and cloud-based emulators. Whether the developer is accessing thousands of devices and run automated tests across multiple platforms via the cloud or utilizing an emulator to duplicate the mobile device experience (hardware and software), the idea remains the same — find bugs and find them fast.
Mobile tests can be conducted manually or as automated tests. Automation is typically used for complex, time-consuming testing scenarios. Testing can also start at the early stages of development, be implemented just before a product launch, or affirm that changes to an existing software are OK to go live too.
Throughout the process, there are many different types of testing for mobile apps, including:Functional
— ensures the software works as designed from the user’s perspective. Even when only adding new features or changing the way your app works, functional testing verifies that the software meets its requirements.
Performance — addresses application performance on different connection types (e.g. 2G, 3G, 4G, or Wifi).
Memory leakage — tests whether memory accessed temporarily to run a program is returned (when it isn’t available memory diminishes until the program can no longer run).
Interrupt — examines how an application reacts to an interruption (e.g. does a game return to where the player was if they leave the app to type a text).Usability
— explores user experience and makes sure that the app is easy to use and will provide customers with a positive experience.
Installation — confirms app installs and uninstalls on a range of devices.
Certification — ensures that applications comply with industry regulations.
Security — validates that the information system of the app protects sensitive data.
Location — considers connectivity changes across networks and locations. Also whether the app is internationalized to account for regional differences.
With so many aspects to test, mobile application developers can struggle to comprehensively test on a tight production timeline. Yet putting the software through its paces under real world conditions — to test not only that it does what is expected but also what happens if a user does something unexpected — can make the difference between releasing a successful product and launch letdown.
Mobile testing challenges
The mobile app environment is changing at breakneck speed. At the same time, individual users are easily frustrated and have many available options to defect to if disappointed. Thus, app developers face many challenges in doing effective mobile testing.
Dealing with device fragmentation in mobile testing
First, there’s device fragmentation. Users today have multiple devices to choose from. Thus, the more successful your app, the worse the problem of device fragmentation can get. Your app needs to play nicely with laptops, phones, eReaders, watches, and more.
How can the average developer hope to keep up? Some 500 million Android devices and 220 million iOS devices have been shipped since 2007. In Cap Gemini’s 2013-2014 Mobile Quality Report, 52% of developer respondents cited lack of devices as a reason they don’t do mobile testing. Even different devices within the same product family can’t be treated as the same as screen size, memory, CPU and hardware are just a few of things that could be different.
More challenging still, is the commoditization of the smartphone, which has led to cheaper mobile devices, usually running some variant of Android, gaining market share. This means that for mobile app acceptance in international and emerging markets, the product must also consider requirements specific to these lower-quality brands.
The breadth of devices available also requires developers to account for platform diversity. Users are often loyal to their mobile device platforms. They are certainly unlikely (with WeChat being a gigantic exception) to pick an app over a platform. So, developers need to be sure that their apps function seamlessly on iOS or Android platforms, which are updated so frequently you may already need an update for your device (even if you just did one before starting this article).
This means that while your developers are coding for the latest and greatest at their disposal, your app needs to account for loyal users who aren’t running the most recent browser or remain loyal to a device that has been around since the Internet’s cave dweller era.
Google Chrome led among mobile/tablet browsers in June 2017 with 55.5% market share compared to Safari’s 33.7% and Android browsers 5.4% — NetMarketShare.com
Knowing the weaknesses in mobile app development makes testing more efficient
Further complicating things, is the range of mobile network operators globally. There is no monopoly on accessing mobile networks (to the disappointment of many heavy hitters worldwide). While this encourages competition that is thought to benefit consumers, for developers it means the app needs to function on networks with different speeds, availability and strengths. Additionally, there are regional differences to account for in trying to release an app internationally. Among the questions that need answered: is your app going to function in local languages? Will it connect seamlessly to popular social networks in that region?
Add to these considerations the fact that web apps will have different compatibility needs than native apps (which are designed for installation on particular devices). Or unexpected windows can pop-up during mobile testing (e.g. a battery warning) and stall the test.
Then, there’s the fact that many of the web professionals behind innovative applications are taking a “mobile last” approach. Without prioritizing mobile user experience, these developers are building apps that don’t translate effectively to mobile devices. They may even be working with tools that are not mobile responsive (such as Photoshop). Thus, while seeking to develop the next best app they are also developing the next best headache for their mobile users.
Mobile testing is complex and challenging. Nonetheless, it is an essential stage in software development success.
Mobile testing with test IO
test IO can connect you with professional testers around the globe with access to the wide variety of devices and operating systems as well as the geographic diversity needed for thorough mobile testing. Offering a breadth of coverage difficult to perform in-house (or through outsourcing), test IO lets you tap into the power of a crowd of experienced, talented testers and access results around the clock and under real-world conditions.
Mobile crowd testing can run the gamut from an exploratory test prior to a new product release to regression testing
to guarantee platform functionality. Plus, our crowdtesting platform removes the managerial headache of coordinating, evaluating, and paying the testers. We’ll take care of the paperwork while your QA teams and engineers focus their attention on continuous delivery of software.
Our on-demand testers engage with your software to help you think about your product in new ways — even uncovering unforeseen issues — while providing clear, usable expert feedback.
Since test IO crowdtesters are not “part of the team” they can’t fall prey to the cognitive biases that even an outsource team develops. They can provide a realistic assessment of how customers will evaluate the software when it’s released.
As they are regularly testing lots of different parts of software, they are also able to find things that an automated script won’t know to even explore. Examples include Facebook logins crashing an app if the user’s profile has restrictive settings or not, or users not being able to click to register on Android mobile keyboards.
Additionally, the input of real-world testers can help push further iterations of your app in the right direction by reducing performance concerns as you continuously adapt your software to work on most mobile devices.
Our knowledgeable testers can help you characterize known defects and discover ones you didn’t yet know about.
Mobile Testing in the Real World: Why it Matters
Mobile devices are intimate objects, never far from users’ hands and increasingly from their thoughts. The average American will spend almost two and a half hours a day using mobile apps in 2017 — and as we know, some of that time is spent in the bathroom, or on the train, or while walking down the street. Increasingly, mobile apps serve as a remote control for the real world, whether paired with a car, a thermostat, or a home security system. In other words, the conditions under which a mobile app operates are about as far from laboratory conditions as it’s possible to be.
Given this level of intimacy, and the gap between how an app behaves in the test lab and how it feels in your hand, controlling your environment, mobile testing in the real world takes on a different level of urgency.