Visual regression tests in the CI workflow: how the layout gets into the pipeline

Modern development teams rely on their continuous integration pipeline: every push triggers a build, the unit tests run through, the linter checks the style, and only when everything is green does the code move into the main branch. This safety net has a blind spot, though. Whether the website still looks the way it is supposed to look after a change is something the classic pipeline does not check. A CSS refactoring can pass every test and still shove the contact form behind the footer.

The reason lies in the nature of the usual kinds of test. Unit tests check functions against expected return values, integration tests the interplay of components, end-to-end tests click their way through workflows and verify that defined elements are present. None of these approaches sees the page the way a human sees it: as a rendered image. A button can be technically present, clickable and correctly labelled — and still remain invisible, white on white.

How such breakages come about is best seen in everyday work: a dependency update pulls in a new version of the CSS framework that sets a few default spacings differently. A refactoring renames a utility class that is still used in one forgotten place. A font file is swapped out and turns out marginally wider, whereupon a navigation bar wraps. None of it throws an error, none of it fails the build — and every one of these changes looks entirely harmless in the code view.

Visual regression tests close this gap with a simple principle. For a defined set of pages a reference state is recorded, the so-called baseline. After every change new screenshots are produced, which a tool compares against that baseline. If the current image deviates, the check fires, and a human decides: intended change or regression?

The interesting question in practice is less the whether than the where: at which point in the CI workflow do such checks belong? Unlike unit tests, which work directly on the source code, visual tests need a rendered page that is reachable over the network. That pushes them to the end of the chain — to the point where a running version of the application already exists.

Services such as ScanU therefore work on a URL basis: you hand over the address of a reachable page, and the service produces screenshots across several browsers and device classes in around 30 seconds — Chrome, Firefox and Safari, each in mobile, tablet and desktop views. How ScanU builds up such a check run step by step is described by the provider itself; the basic pattern holds for every URL-based tool, though: without an environment that serves the new state, there is nothing to photograph.

This fits most elegantly into workflows with so-called deploy previews. Many hosting platforms automatically build a dedicated preview environment with its own address for every pull request. That address is precisely the natural point of attachment for the visual comparison: the preview shows the state after the change, the baseline the approved state before it. If the comparison reports a deviation, everyone involved sees the problem before the code reaches the main branch — and the discussion takes place where it belongs: in the review.

In concrete terms the sequence in such a pipeline looks like this: first the CI builds the new state, then the hosting platform publishes the preview environment and reports back its address, and only after that may the visual check step start. This is exactly the point at which something goes wrong most often in practice. Anyone who triggers the screenshot run before the preview has been fully deployed photographs a half-loaded page or an error message — and then compares it, in all seriousness, against the baseline. A quick availability check of the preview address before the actual run spares you that sort of confusion. Once the report is finished, its link belongs on the pull request, as a comment or in the description, so that the assessment happens where the change is being discussed anyway. Shareable reports, such as the ones ScanU produces, are meant for exactly this move: the report is the shared basis for discussion, not a tool window that only one person has open.

Teams without preview environments fall back on the staging environment and check after the merge but before the production deployment. That is the second-best solution: regressions surface later, once several changes have already been mixed together, and the hunt for the culprit becomes more laborious. An additional regular run against the production site also catches changes that do not originate in your own code at all — for instance when an externally embedded script or a CMS update shifts the appearance.

The technical integration is the smaller part of the work. The larger part is a process matter: maintaining the baseline. A baseline is not a technical artefact but an agreement — this is how the page is supposed to look. Anyone who rolls out an intended design change has to update the baseline deliberately afterwards, otherwise every subsequent run reports the same expected deviation. Anyone who instead updates it reflexively after every alarm will sooner or later declare a genuine fault to be the desired state.

It has proved sound to treat baseline updates like code changes: someone looks at them, someone approves them, and the approval is traceable. ScanU displays the baseline and the current state side by side for this and generates shareable reports. That sounds unspectacular, but it changes the collaboration: colleagues from design or product management can also judge a deviation without starting a development environment — and the decision “intended or broken” is often not a purely technical one.

A team should also clarify who owns the check. Visual tests that are assigned to nobody reliably fall into neglect: reports pile up, nobody feels responsible, and after a month everyone ignores them. The same rule as for broken builds has proved itself — whoever caused the change deals with the reported deviation, and a rotating duty keeps an eye on the regular runs.

Planning also includes the question of how often to check. Not every commit needs the full pass over all pages and device combinations. Billing models such as ScanU's credit system make that consideration concrete: the free tier covers 500 credits a month for one project, while the paid tiers range from 3,000 credits for 19 euros up to 50,000 credits for 49 euros a month. Anyone who drives every pipeline run across every device profile burns through their allowance quickly — and incidentally produces more reports than the team can even look at.

A staggered approach makes more sense. Pull requests with changes to styles or templates get the comparison across the most important pages; a daily or weekly run covers the broader set across all browsers; before a release everything runs once. That keeps the signal-to-noise ratio healthy, and every alarm has a real chance of being taken seriously. A visual test whose messages the team routinely clicks away is worth less than none at all.

When selecting pages the rule is: templates rather than instances. An online shop does not need to photograph 2,000 product pages, but one representative page per layout type. The home page, one product or article page per template, the basket, forms and the error page cover the bulk of the risk in most projects. Pages with high business value — the checkout, say — deserve more device combinations than the Impressum, the legally required site notice.

The same trade-off applies to device profiles. ScanU's price tiers also differ in the number of available devices — from ten in the free tier up to 98 in the largest. That tempts you towards completeness, yet more profiles mean more images, more credits and more reviewing effort. More workable is to take your own traffic figures as a guide: the three or four device combinations through which the majority of visitors actually arrive, plus one representative of each of the extremes — very narrow, very wide.

One hurdle deserves particular attention: dynamic content. Rotating teasers, personalised recommendations, cookie banners, displayed clock times or advertising change the image on every call — entirely without a regression. Such false alarms are the most common reason why teams switch visual tests off again after a few weeks. The remedies are well known: test environments with fixed sample data, disabled animations, consistent starting states. False alarms never disappear entirely, but they can be pushed down to a tolerable level.

Honesty also means naming the limits. A screenshot comparison that passes says nothing about whether the form actually sends data, whether the search returns hits or whether the checkout runs through. Visual tests check presentation, not function — they complement unit and end-to-end tests, they do not replace them. Anyone who reduces their functional test coverage in favour of screenshots is trading one risk for another.

The significance of the result also depends on the environment. URL-based checks photograph whatever the given address serves up. If the staging environment differs markedly from production — different data, different feature flags, missing third-party scripts — you are checking a distorted picture. And areas behind a login need separate consideration before an external service can capture them at all.

For the check to stay anchored in the team, a sober approach to its results helps. It is worth taking stock occasionally: how many reported deviations were genuine regressions, how many false alarms, how many intended changes? If the ratio tips towards false alarms, that is not a reason to switch off but a work order for the test environment. And if the comparison simply finds nothing over months, a team may also lower the check frequency — test discipline does not mean maximum build-out but appropriateness.

An underrated aspect is history. If a regression is only noticed days later, a look back helps: when did the page last look correct, which period comes into question as the cause? ScanU keeps histories for between three and 90 days depending on the price tier. That belongs in the planning too — a three-day history is quickly used up over a long weekend, whereas 90 days also answer the question of whether a problem existed before the last quarterly release.

For getting started there is much to be said for a small beginning: one project, a handful of pages, a weekly run plus a run per release. As confidence in the alarms grows, closer interlocking with the pipeline follows. Details on how it works can be found in ScanU's documentation; an overview in English is offered by the provider's English-language product page.

That leaves the fundamental question: is the effort worth it? The answer depends less on team size than on the frequency of change. Anyone who deploys weekly produces weekly opportunities for silent layout breakages — and, without systematic checking, discovers them only once users have long since seen them. The CI pipeline has taught everyday development to catch logic errors before release. There are few reasons to deny the most visible layer of a website the same protection.