Visual regression testing: definitions and tools

Quick, I’ll give you a dollar if you can spot all the differences in the two images above! I’ll give you a hundred dollars if you can spot all the differences in one minute. Every second it takes over one minute, I’ll deduct a dollar. And if you miss anything, that’ll cost your team two hundred bucks.

Does that pressure sound familiar?

What is Visual Regression Testing?

Visual regression testing verifies that any new visual aspect of a piece of software doesn’t disrupt existing elements. In other words, it checks to make sure that a new design element won’t break other components in your product. 

A visual regression test compares screenshots taken before and after code changes, so you may have heard them referred to as “visual snapshot tests.” It’s a visual testing tool that would’ve made those “spot the difference” games you played as a kid a lot easier.

How Visual Regression Testing Applies to Other Agencies

For Fast Moving Agencies (“Move fast and break things”):

You’ll make the change, trust that your CSS is well-structured, and have a quick look at the site. You might miss that the “mouse text” class got improperly re-used only in the hamburger menu at the tablet breakpoint. But don’t worry—your Executive Vice President will find that problem two weeks later when they discover that the navigation is broken on their iPad Pro. Whoops!

For Measured Agencies (“Belt and suspenders”): 

You’ll prepare the change and kick off a round of Visual QA. Your QA team has many automated tools at their disposal, but the tools they use only test to see if items are present or missing on a page or that the items do the right thing when clicked. The only way to see if lines are wrapping awkwardly due to an unexpected text size change is to have a tester look at every page of the site. And every time they find something that might not be right, they’ll raise an “observation” to be discussed with design at the next standup. You deploy the text size change with no errors… a month from now.

Using Visual Regression Testing Tools to Catch Changes

I’ve fed both versions of the image to a computer and instructed it to highlight all the differences in magenta. Did you catch all of the changes?

• The missing desk lamp?
• The addition of text on the closest blank page?
• All the color changes, including the blouse that changed from beige to a light pink?

Did you catch that the entire second image shifted one pixel to the left? This will be entirely undetectable for a human… until the user shifts to a dark theme. Suddenly the image has an unexpected white hairline all along one side!

A visual regression testing tool catches all those changes in one step and makes them easy to spot.

Here’s How We Use Visual Regression Testing Tools at Think Company

Backstop for Visual Screenshot Testing

First, we use a visual testing tool called Backstop to take pictures of entire web pages, and then compare those pictures to a different set of pictures. Backstop provides a “cool scrubber thingy” to show how those snapshots are different. Here’s YouTube user Garris Shipon demonstrating how you can “scrub” across a page to make changes crystal-clear:

Backstop can turn a week-long hunt for visual differences into a ten-minute “push the button” exercise. No magenta? No changes. No changes? No problem—SHIP THAT SITE.

Additional Visual Regression Testing Tools

Second, we make visual regression testing a part of our development workflow at the component level. Here’s the set of tools we’re using right now—Jest for writing tests and Puppeteer for making screenshots—so that every pull request compares every component on every commit.

Using visual regression testing as a part of development means that “Whoops, I accidentally made a visual change seventy-five pages away, in a breakpoint that shouldn’t exist” is no longer something we worry about. Those (inevitable!) problems surface right away while the developer is working. They’re surfaced right in front of the person who can do something about it instead of waiting two weeks for an observation from the QA team.

Learn More About Visual Regression Testing for Your Agency

Using visual regression testing both as a part of your QA process—and as a part of your continuous integration flow—reduces stress and errors, makes Pull Requests easier to review, and (dare we say it?) puts a lot of the fun back into hammering on the front-end. And all of this happens without hammering front end regression testing. Think Company offers workshops and hands-on clinics to show you and your agency how Think Company does it and teach your team to customize these tools to match your flow. Let us know if you’re interested in learning more!