Testing Solutions

Beta Testing

The success or failure of your app in the market will be determined by real people. Gain crucial insights before you launch by beta testing with real users under real conditions.

What is Beta Testing?

Beta testing answers the most important questions at the end of the development process: does this software meet customers’ expectations, and should we ship it? Often called User Acceptance Testing, beta testing helps organisations launch products successfully in the marketplace.

Beta Testing with test IO

Beta testing has gotten a reboot. With the rise of agile, iterative development patterns, and lately with the rise of continuous delivery, the old idea of a “beta testing phase” — where product development stops while customers chew on on the software — is no longer applicable. Yet there’s no doubt that beta testing, or user acceptance testing, remains an important part of the software development success. Here are some ways to adapt contemporary beta testing strategies for your organization.

What is beta testing today?

Contemporary beta testing arises from the theory of agile development and the advent of SaaS/cloud deployments. As originally expressed by its proponents, agile development prized iterative development in concert with your customer — giving development teams more or less immediate feedback on changes that they made, from people actually in a position to provide constructive input.

Even as they’ve adopted iterative development methodologies, this is not how the real world works for most teams that develop software for external customers. Unless you’re very fortunate, your customers don’t have the time or inclination to beta test your pre-release software. Even if they did, they would be unlikely to do so within the rapid timeframes called for by agile practice — or by demanding management teams whose knowledge of agile practice is limited to, “Ship more stuff faster.”

So, in practical terms, what “beta testing” is has changed greatly over the last 20 years. Broadly speaking, there are four strategies.

The four ways of beta testing

Whole Product, Whole Community Beta Testing

In the Web 2.0 era about ten years ago it was common for companies to release their software to everyone with the word “beta” on it to connote that they weren’t entirely confident in it. Gmail, for example, was famously “in beta testing” for five years. This practice has receded in the mobile era, as the beta label can discourage app store downloads. It can still work for entirely new products, but as most teams work most of the time on new features that improve existing products, it is not broadly applicable.

Partial Product, Small Community Beta Testing

Today, big companies such as Google or Facebook typically release new products or features to a subset of users after thorough internal testing has already been done. With a billion users, the software development teams at these companies can release new features or modified features to 0.1% of their user communities and still gain quantitative, qualitative, and telemetric feedback. This is a tremendous competitive advantage for large companies. Many companies lack the traffic and the technical and organizational capabilities to perform beta testing safely with a subset of their users. It can also be a hard sell when revenue is on the line — if Facebook’s engagement goes down for 0.1% of their users, they’ll probably be fine. The same might not be true for a smaller company whose customers pay them directly.

Whole Product, Small Community Beta Testing

This is the traditional form of beta testing. Perhaps you’ve been a participant in a beta test like this for an eagerly anticipated version of Apple’s iOS — it’s scary to put beta software on your phone, but with Apple there’s the thrill of the new, and you want to be among the first to see how the new OS works! And that’s the trick. Whole product, small community beta tests work best when you can identify a user community that’s passionate about your product, and is strongly motivated to give you good and timely feedback. Make no mistake, though — conducting such a test is a logistically complex undertaking. You have to find these customers, distribute software to them, support them when they have trouble, and collect and synthesize their feedback in an actionable and timely way. Few companies have the resources and customer commitment to conduct whole product, small community beta tests.

Partial Product, Whole Community Beta Testing

When you add a new feature to an existing product, it’s sometimes possible to perform beta testing on that feature alone, even with your entire user community. This works best when you have a feature that’s both new and well-separated from the rest of your product. You have the opportunity to get excellent telemetry and, depending on your user community, quantitative and qualitative feedback. However, this approach is risky if the new section is not discrete from the primary product, as there is a potential for confusion, and defections or increased support costs, among your customers.

Why beta testing is needed

Let’s take a step back for a moment and ask why we’re talking about beta testing in the first place. Also known as User Acceptance Testing (UAT), beta testing at the end of the development process offers crucial insight into how customers will interact with the software and whether the product will meet their expectations. In other words, beta testing helps answer the crucial question: “Should we ship it?”

The cost of shipping bad software is increasingly high. There are more than 940,000 applications across 2.1 billion devices, per Flurry Analytics. This means that when someone downloads your app, you have only a couple of minutes to convince them that this is something they need to use on a regular basis. In this short time users need to understand how the app will help them accomplish something they value, according to John Egan, Pinterest’s growth engineering manager, in a Flurry Analytics article.

Beta testing, if done well, can help you find bugs and work out the kinks before going public and making that critical first impression.

After all, a software can function just fine, but if poor product design causes user friction or frustration, they’ll abandon your app. Localytics 2016 research tells us simply acquiring a user isn’t enough — 25% will use the app only once — retention is the key. Even that is challenging with 80% of all app users churning within 90 days. Of course, the truly relevant metric for you will be user churn related to your competitors, yet the danger of underestimating UX stumbling blocks is clear.

Insights you can only get by beta testing

Beta testing, though, provides insights not only into functionality but also into user experience. This offers a final vote of confidence on user acceptance of the product while in its First Customer Ship (FCS) state. Instead of relying entirely on how the software performs in the lab, beta testing lets you see whether the same level of performance is reached in real world settings. After all, the product will need to succeed in thousands of different environments in as many different use scenarios.

By taking the software out of the hands of the developers who are closest to the product, beta tests can identify elements of functionality that were overlooked in the lab. Plus you might gain feedback that helps you improve feature versions of the product or suggests new products entirely.

With beta testing developers can affirm quality, usability, stability, security and reliability. There is little margin of error today. Unresolved issues or bugs or undiscovered functionality needs, may keep your app from working well when it’s released. Meanwhile, your user will quickly move on to another app providing the same feature seamlessly.

Optimizing beta testing

Some Quality Assurance (QA) managers are not fans of beta testing. They resist product or marketing team beta testing because they end up with a pile of poorly described issues. Without talented people in place to wrangle the customer feedback communication and liaise with product or marketing teams, developers can also find it difficult to prioritize addressing customer concerns and functional defects.

In today’s social media savvy environment the software developer might also worry about the risk of someone sharing beta defects with a wider audience. Additionally, beta testing can be expensive to administer while slowing the release of the software.

Basically, when you put pre-release software in front of beta testers, you’re trying to find the optimal combination of:

Speed – how quickly and reliably you can get results.

Quality of explicit feedback – or more specifically, the ratio of useful feedback to noise.

Quality/quantity of telemetry – logging what’s truly going on in the app for developers to effectively troubleshoot and product managers to see what users are actually doing.

Protection of revenue – making sure poor or incomplete products don’t cause you to lose money.

Improvement/protection of customer relationships – does “opening the kimono” to your customers help or hurt those relationships?

Prioritize these parameters, and then map them to your organizational capabilities. When beta programs fail, it is often because the program owner lacks the institutional support required to recruit the testers, so if your company is well set up for small-community, partial product beta testing you’ve eliminated that problem. Otherwise, it’s likely that your biggest problem will be recruiting testers and managing their feedback.

One solution? Seek expert help.

Beta testing with the Crowd

Crowdtesting firms, of which test IO is one, can quickly convene a user population that mirrors your target customer base. But these proxy customers are testers who are experienced with pre-release software: logging bugs, pointing out missing functionality, or identifying absent error conditions and other user interface glitches that could torpedo the success of your launch. Plus, they are savvy about documenting their finds in a clear, concise fashion that is useful — not noise.

Some estimates report that only 20% of Beta testers send back useful information. A better outcome of beta testing is possible with the crowdtesting expertise of test IO. You don’t need to be the size of Google or Facebook to get useful feedback and benefit from the feedback of trained, professional testers.

In this sense, the crowd levels the playing field against companies with legions of eager beta testers (like Apple) or users who become beta testers without even knowing it (like Google). We connect your company to the testers who can interact with your software under real-world conditions — considering mobile device fragmentation and location diversity — and provide documentation that helps your developers better understand what needs to be done next.

test IO can connect you with professional beta testers around the globe with a variety of technical expertise, educational backgrounds, language skills, geographic locations and more. If recruiting testers in the available time frame is your biggest challenge, a crowd of testers can test your known defects as well as your yet-to-be-discovered ones and improve the quality of your release. test IO also takes away the logistical problems of communicating with the testers and distributing pre-release apps to them, even iOS apps. Try achieving best practice of beta testing by trying out test IO’s professional crowd of beta testers now.

Basic tips for best beta testing, however you do it

To accomplish a successful beta program it helps to:

1) Set clear, achievable goals
Obviously, you want product feedback. But one criticism of beta testing is that it generates a lot of noise. Combat this by planning in advance, for instance, whether you want to focus on user interface, scalability, or performance. Determine what use cases or scenarios will be most useful when tested. If you try to accomplish everything in a beta test — or approach the test just looking for “general feedback” — you are unlikely to make as much progress as your competitor who has set out to test specific goals.

2) Specify expectations
Before starting beta testing, set parameters delineating the testing’s scope and duration. Determine an intended testing team size and clearly outline the stakeholders and their responsibilities. Plan ahead to accomplish the most effective test by balancing the moving parts of your testing team size with the duration of the test and your specific objectives. In beta testing, a long period of testing with actual customers while the production code is still available to most is best practice. This is in an ideal situation, though, where the customer is savvy and capable of communicating useful feedback.

3) Think like a user
Beta testing is meant to put the product through its “real world” paces. This means testers want to be creative. Think not only about what the smart user might do if using the software exactly as planned but also consider that renegade user who doesn’t read a single one of your well crafted instructions and wants to see what the software does when he or she goes rogue. You’re not going to be able to control how your customers use your product — so don’t limit the testers to a predetermined, narrow view of how the product should work.

4) Identify the right audience for results
Effective beta testing generates a wide range of feedback. Not every member of your organization needs to be informed of the entire deluge of results. Distribute feedback sparingly, thinking first about the value of what you are sharing to the person receiving it. For instance, new business or product planning staffers will be more interested in feature requests while developers want to know what isn’t working for users and marketing will want to hear the enthusiastic testimonials.

Ship Faster, Sleep Better

Get a Demo