The Equifax Hack and Software Testing: A Cautionary Tale

I don’t have any inside, or even outside, knowledge of this question. Equifax isn’t a customer of ours, and I’m no particular expert in Apache Struts 2.

Equifax may be an unusually lax organization in possession of an unusually rich trove of personal data, but the problem they faced in patching their application is one familiar to anyone responsible for a software application over multiple years.

On the face of it, this isn’t immediately obvious. The software that’s running the Voyager spacecraft is working well 40 years later, billions of miles from home. But modern software sits atop a stack of frameworks and layers of abstraction that complicate maintenance even as they simplify development. Nobody would want write the Equifax web app in assembly language (okay, maybe somebody would). Frameworks like Struts 2 are certainly preferable to writing your own web app framework in Java. But once you’ve made that bargain, you need to worry maintaining all the things that go into Struts: the framework itself, the JDK, the app server, the operating system, and so on.

Some of these things are relatively easy to patch or update. OS patches generally require a server restart, but that’s not so terrible, and a SysAdmin can handle it. Once you’ve patched an application framework, though, you need a programmer, and once you’ve changed a program, you need testing. Cribbing from Peter Bright at ArsTechnica, the sequence might look something like this:

"You might have to dig out older JDK versions to get it to build, find an old copy of an old internal JAR that's somehow gone missing, all the usual problems that happen when you try to rebuild an old application. That's assuming, of course, that you have the source code and build scripts, and that alone is far from guaranteed. I bet that there will be developers who find that the version in source control for some reason doesn't quite match the version that's deployed, or that they have no source at all, or that it doesn't build for whatever reason.

So, your developers have to update their Maven or gradle or (god forbid) Ant build scripts and bump the version number for the Struts dependency to grab the new version.

You then have to hope that nothing is broken. If you're using Struts 2.3.5 then in theory Struts 2.3.32 won't break anything. In theory it's just bug fixes and security updates, because the major.minor version is unchanged. In theory."

It turns out, though, that the theory wasn’t exactly true here. According to a developer who did a number of these patches, “[The] upgrade from 2.3.24.1 to 2.3.31 is a completely breaking change (like, no pages of the app will display) due to a Tiles issue. So you probably can't just upgrade.”

You might have to change a few other things in your code just to get the application to work at all after the upgrade. And then, how do you know that the application actually works?

That’s where testing comes in, and testing this kind of thing is difficult. If you did everything you were supposed to do, your application has excellent unit test coverage and all of your unit tests will pass. Unit tests will only get you so far, since by definition they’re designed to test atomic units of code in isolation. What you’ve changed is more like the environment your code runs in. So you also want to perform integration tests and functional tests, and -- depending on the patch and the application -- non-functional tests like performance tests and security tests as well.

This means that you need people to spin up a test environment to run the application, and people to execute the actual tests. In these days of containerization and test automation, you might imagine that this is trivial. First of all, some applications were built before our current enlightened age, and second of all, things that are automated don’t always age well either. Making sure all of your scripts still run correctly after several years is not trivial. Almost no one has automated 100% of the applicable test cases, or even written 100% of the applicable test cases in English. So you will probably also need some testers to run the tests, and to figure out what tests haven’t been specified but still need to be run.

I say none of this to exonerate Equifax. They screwed up massively, and they should pay a heavy price for their negligence precisely because the data they hold is so valuable to criminals. They should be exemplary in how they manage security.

But almost any company that ships -- and therefore must maintain -- software faces many of the same challenges. Nobody has an infinite number of people or an infinite amount of time, and everyone has to patch and update software sometime. Here are some ways that we can protect ourselves when we need to do this in the future:

• Invest in deployment -- especially for apps you’re not deploying that often
  These days, it’s become easier to version, maintain, and deploy all of the resources required to run an application. It’s much easier to stand up that application in environments that closely mirror production. If you haven’t made the changes to enable this magic, you’re probably not going to start doing so precisely when you’re doing fewer deployments and less development of a particular application. This is a mistake. The further you get from everyday development of an app, the fewer people in your organization will remember all the little tricks required to stand it up. If an app is going into maintenance mode, it needs to be really easy to deploy.
• Have a testing plan in place for applications you’re maintaining
  The testing plan needs to include human resources for performing the parts of the testing that are not automated. One of the benefits of crowdtesting is that you always have people on standby to test your application when you push an update. Of course, crowdtesters are only part of the solution. They need a test environment and a set of directives to operate effectively.
• Stay in shape by testing and deploying
  Deployment and testing are like any other habit: if you keep doing it, you remember how; if you stop, it’s hard to start again. At test IO, we see this pattern often. People slow down in updating their mobile apps. When there’s a significant OS upgrade (stay tuned for iOS 11, now without a home button!), they scramble to recreate processes that used to work. Then they find a bunch of problems that their customers had probably been seeing for months. If you make deployment and testing simpler, you’ll probably make more incremental updates. This will keep your software in better shape and keep you ready for when an update really is critical.

I know this sounds crazy, but some updates are small enough that you can push them into production and then test them. We have customers who do this with test IO’s crowdtesters. We get a trigger when something’s changed in production. Our testers are then the first on the scene to make sure everything’s running okay.

Does this sound risky to you? One one level, of course it does. If you’re balancing it against not deploying updates because you believe the testing burden is too high, you may be thinking about risk the wrong way. Sometimes the biggest risk is to do nothing. Just ask Equifax or any of the 147 million people affected by their negligence.