Squashing UI bugs through visual regression testing

Your app passed all of its automated tests and even functional tests. But anyone using a smaller iPhone now sees an awkward misalignment and a bunch of scrunched up text—not to mention blurry background images and icons that don’t properly scale. And all those users you thought abandoned IE 11 for Microsoft Edge now see nothing but missing images!

What kind of testing is that? You might want to try visual regression testing instead.

What is visual regression testing, you ask? Visual regression testing is a way of taking screenshots of the site you are developing. These screenshots are often taken before and after you update your front-end code.

When I say front-end code, I'm referring to things like CSS, Javascript, and HTML. You can take the screenshots at any stage, whether it’s testing, staging or production. Afterwards, you can compare previous and current screenshots to see if your last updates have broken anything.

Over the years, various frameworks have emerged to help developers build cross platform apps in one shot. Frameworks such as React or Bootstrap have advanced the way we build technology. Visually, these frameworks take your grid layout into account across platforms.

When you use these frameworks, you are building for both web and mobile at the same. You also have tools like Xamarin, which uses C# to allow developers to write native Android, iOS, and Windows apps. These all make developing a lot easier than it used to be. But when it comes to properly testing your products in the end, visual regression testing is important to check if what you've created really does look equally good on multiple platforms.

The Nielsen Global Digital Landscape report “Screen Wars: The Battle for Eye space in a TV-Everywhere world,” asserts that around the world:

• 76% enjoy the freedom of being connected anywhere, anytime
• 69% think face-to-face interactions are being replaced with electronic ones
• 63% believe bigger is better when it comes to screen size
• Mobile phones are the most commonly cited go-to device for on-the-go viewing

Functionality will always be a key factor in why someone uses your product or service. But it’s important that your user interface is easy to use, too. With so many users on different platforms, you have to maintain product quality across platforms.

Depending on the user's screen size and resolution, an interface can change dramatically from screen to screen. Screen size is generally measured as the diagonal physical measurement of the screen in inches. The screen resolution is the number of pixels on the screen, displayed by width and height.

For example, the screen size of an Android LG G4 is 5.5 inches. The screen resolution is 1440 pixels wide and 2560px high. You can view screen and resolution sizes for various platforms here.

An important thing to know about visual regression testing is that it mostly focuses on the visual—hence the word “visual” in the name. Reviewing characteristics like layout, missing images, and blurry icons is a very important part of the process.

Be sure not to confuse visual regression testing with regression testing. They are similar in that they both verify that your previous developments still behave as expected. But regression tests are normally done after the development of your product is complete.

Regression tests can also check for a number of other things, such as product enhancements, fixes for an existing bug, and configuration changes. Most teams create test cases based on product guidelines and standards. If they are working in an agile environment, teams are most likely running tests based on user story acceptance criteria.

If your team has the capacity, consider using both visual regression testing and regression testing. You’ll be able to effectively squash front-end and back-end bugs.

Now you’re probably asking yourself, “How on earth am I going to test this?” Well, you have options for automated and manual testing. There are a ton of automated options that allow you to take screenshots. It can be surprising how many visual regression testing tools are out there.

• Percy stores the original DOM snapshot and page assets. Afterwards, they render the same page with different widths. You provide them with the breakpoints if you are testing for both mobile and web. The downside of Percy is that it doesn’t render Javascript. So if any of your changes rely on Javascript they might not show up. The website does mention, however, that you can contact them for workaround options.
• Ghost Inspector tracks what your website is supposed to look like via their screenshot capture feature. They take a screenshot at the end of every test run. They review pixel by pixel to see if there are differences from the last screenshot. You can set a tolerance from 0%-90%. If the screenshot has changed more than that, it will flag a failure. The downside of Ghost Inspector is that the system compares a maximum area of 1600 x 20,000 pixels. This means any content further down than 20,000 pixels won’t be compared.

There are also other tools like PhantomCSS, PhantomJS, CasperJS, and Gruntost, which are command line tools. You can install all of them through the Node.js package manager npm. Essentially, you’ll be writing scripts to use them.

Reviewing the pros and cons of automated visual regression testing, I’ve noticed some common issues with animation and dynamic data. Automating any dynamic data, such as a date/time change, can create a fail scenario since it picks up a difference between your previous and current screenshots.

If you have any animation on your website, you’ll have to find a way to freeze animated gifs on a specific frame. Or if the tool allows it, you can try to freeze the CSS animation and transition styles.

This is why I recommend adding humans as well. You can do this with crowdsourced testers on real devices. With a fast-changing UI, it may become difficult to do things such as freezing animations frame-by-frame while also trying to develop.

A crowdsourced tester can help keep an eye out for visual things you should focus on. They can also limit the false negatives and positives you’ll see by testing on real devices. The really great thing about crowdsourced testers is that you’ll have access to different users across the world on various platforms. These are essentially the same users of your product once you go to market. That’s why it can be helpful to combine both automated and manual testing.

In the famous words of Martin Leblanc, the founder of Iconfinder, “A user interface is like a joke. If you have to explain it, it’s not that good.” People shouldn’t be in the bathroom or the boardroom trying to figure out your UI—that’s just cruel.