Visual Regression

A new breed of tools have started appearing on our radar supporting the very idea of automated visual regression. A visual regression testing tool performs front-end or user-interface(UI) regression testing by capturing the screenshots of web pages/UI and compare them with the original images (either historical baseline screenshots or reference images from live website). If new screenshot deviate from baseline or reference screenshot, visual regression tool will warn you. In context of web applications, visual regression tools are performing regression test on CSS changes.

It is important to note that screenshot based visual regression testing can only work when UI behavior is predictable. Mutable UI components should be normally excluded from visual regression. Visual regression tools works really well for testing the web applications, responsive layouts etc.

Visual regression tools also provide simple image diff functionality which compares new and original image and outputs a third diff image. For image comparison these tools rely on the image libraries such as ImageMagic, Python Image Library(PIL), Resemble.js etc. This functionality is a key differentiator for visual regression tools.

For a more fine level image comparison - smallest pixel or slightest color change, tools like Kaleidoscope¹ can be used. Tools like Kaleidoscope makes reviewing screenshot files much easier and quicker.

For screenshot capturing visual regression tools either use Selenium WebDriver or a headless browser such as PhantomJS. Visual regression tools are DevOps friendly and can be used with your existing continuous integration and delivery setup. As part of process, screenshots can be checked into your version controlled repository. It can allow visual review of the UI changes as part of your existing code review process.

To this end I have found three visual regression testing tools those fits above mentioned criteria,

Wraith

Wraith is a screenshot comparison tool, created by developers at BBC News ². It compares screenshots from live website to staging or test environment³. Wraith uses ImageMagic to compare images. Wraith allows to vary the resolution and enables to test breaking point.

####Huxley Huxley is developed by Facebook team for visual regressions in web applications⁴. It compares captured images to baseline images.

####PhantomCSS PhantomCSS⁵ automates visual regression testing with PhantomJS and Resemble.js⁶.