Screenshot diffs in practice: baselines, thresholds and false alarms

On paper a visual test is the simplest kind of test in the world: two images, one comparison, done. Anyone who tries it soon notices that between “compare two images” and “know reliably whether the website is broken” lies an entire craft. This craft has three core concepts — baseline, threshold, false alarm — and anyone who understands them also understands why visual tests are indispensable in some teams and switched off again after three weeks in others.

At the beginning stands the baseline: a set of screenshots that records the intended state of the page. What matters is what the baseline means — and what it does not. It is not an objective measure of correctness but a snapshot that somebody has declared to be the desired state. A diff against the baseline therefore never says “the page is broken”, only “the page is different from the approved state”. Whether different also means worse is something no algorithm can decide.

The comparison itself works pixel by pixel in principle: for every image position the colour deviation between the baseline and the current screenshot is determined, and the sum of the deviations yields a measure of the change. This basic principle is robust and fast — and it has a built-in weakness: it treats every deviation alike, whether a basket button has disappeared or a font has been smoothed marginally differently.

That is exactly where the biggest practical problem of visual tests comes from: noise. Font smoothing turns out differently depending on the system and the rendering run. Edges are drawn a subpixel earlier one time, later the next. Images go through compression that produces minimal colour deviations. None of it is a regression — but all of it produces pixel differences. A naive comparison that raises the alarm at every differing pixel therefore reports something practically every time.

The second source of noise is the content itself. A displayed clock time changes on every call. A carousel never stands on the same image twice. Advertising slots deliver changing motifs, recommendation bars changing products, a cookie banner appears sometimes before and sometimes after the screenshot. Anyone photographing a news site in the morning and at midday gets two completely different images — and both are correct.

The tool against the noise is called the threshold: only when the deviation exceeds a defined level does the comparison count as conspicuous. With that begins a balancing act that cannot be optimised away. A strict threshold reports every trifle and creates alarm fatigue; a generous one overlooks genuine faults — a price that has slipped can stay below the threshold if it affects only a few pixels. There is no single right value; there is only the fitting value for a particular page and a particular team.

For setting it there is at least one tangible procedure: measure your own noise. To do so you photograph the same page twice in immediate succession, without changing anything at all, and compare the two shots with each other. Every difference this comparison reports is by definition noise — font smoothing, compression, timing. Anyone who repeats this measurement for the most important pages and device combinations knows the base level of their own test environment and can set the threshold just above it instead of guessing at it. Incidentally, the procedure delivers an early warning signal: if the base level measured this way rises noticeably later on, the environment has become more restless — and it is worth pursuing the cause before the alarms lose their credibility.

Alarm fatigue is not a soft side topic here but the most common death blow for visual tests. A checking system that goes off three times a day for no reason will be taken seriously by nobody after two weeks — and then the alarm that actually reports a broken payment page is just one of many that get routinely clicked away. A visual test the team does not trust is worse than none at all, because it feigns a security it does not deliver.

That is why it is worth working on the general conditions before turning the threshold dial. Fixed sample data instead of live data in test environments, disabled animations, consistent starting states — an already-answered cookie dialogue, for instance — sufficient waiting times until the page has fully loaded and identical window sizes for every run have all proved themselves. Tool-specific pointers on this belong in every introduction; for the service considered here they are collected in ScanU's documentation on the provider's website. Each of these measures removes one source of noise — and makes the threshold less important, because genuine and false alarms separate more clearly.

For stubborn sources of noise — the advertising slot, the live date, the section with user comments — some tools additionally offer the option of excluding areas of the image from the comparison. That is effective, but it has a price you should know about: an excluded area is a blind spot. Anything at all can happen there in future without anyone reporting it. The trade-off is therefore not “exclude or endure false alarms”, but: is this area important enough that you want to hear about changes to it? For a rotating advertising banner the answer is usually no, for a price column always yes. Defined exclusion areas should also be documented and reviewed occasionally — otherwise silent patches accumulate, and the checking covers less than everyone involved believes.

A related decision concerns the image section. A shot of the visible window area checks what visitors see without scrolling — fast and economical, but blind to everything below it. A shot of the full page length captures everything, but reacts more sensitively in return: if an element in the upper area grows by twenty pixels, the whole rest of the page shifts downwards, and the comparison marks areas whose content has not changed at all. For compact, stable pages the full length is usually the right choice; with very long, lively pages it can be wiser to check the critical sections specifically. That too is not a technical trifle but a decision about which faults you want to see — and which you knowingly accept.

The second big methodological topic is maintaining the baseline over time. Websites change on purpose: redesigns, new content, seasonal campaigns. After every intended change the baseline has to be brought along, otherwise every future run reports the same expected difference. The danger lies in routine: anyone who sees baseline updates as an irksome click will at some point update reflexively — and thereby, sooner or later, declare a genuine fault to be the new desired state. From then on every green run confirms the broken version.

A simple principle helps against that: baseline updates are approvals and deserve the same care as a code review. Someone looks at the difference, someone decides deliberately, someone can retrace the decision later. Tools support this to varying degrees; ScanU, for instance, displays the baseline and the current state side by side and generates shareable reports, so that non-technical people can also judge a deviation — which comparison functions are available in detail is shown on the service's feature page. The principle, though, is tool-independent: approving a baseline is a substantive decision, not a technical one.

One dimension is readily glossed over in introductory texts: time. Regressions are not always noticed straight away. Then the search begins: since when has it been like this? Which change comes into question? This is where history pays off — the ability to look at older states and narrow down the moment of the change. With ScanU the retention depends on the price tier, from three days in the free tier up to 90 days in the largest. For the methodology that means: anyone who checks only rarely needs histories reaching further back; anyone who checks daily narrows down periods well even with a short history.

That leaves the most fundamental limit of the procedure: the semantic gap. A pixel comparison does not know what it is comparing. Two pixels of shift on a payment button can be a symptom of a broken layout grid; twenty pixels of shift in a footer are usually irrelevant. The diff does not know the difference — it measures area, not meaning. The last word therefore always belongs to a human who knows the page and can judge which deviation has business relevance.

And finally there is what a screenshot diff fundamentally cannot see: everything that does not show up in the image. A page can look faultless and still display wrong data, a form can be perfectly rendered and fail on submission, a server can serve outdated content correctly. Visual tests are one layer in the fabric of testing — alongside functional tests, monitoring and occasional human curiosity — not a replacement for it.

How the individual building blocks fit together is shown by a worked-through example — not a case study, but a typical sequence. Week one: a team creates baselines for ten pages and finds dozens of reported deviations the next morning. The review shows that almost all of them go back to a carousel and the cookie dialogue. Week two: calm the test environment — fixed sample data, a defined starting state, animations off; the runs go quiet. Week three: a dependency update shifts the spacing of the navigation in one of the three engines — the first genuine find, which nobody would have noticed before, because in everyday work nobody routinely opens all the engines. Week four: a redesign of the home page produces expected differences across all views; after review the baseline is deliberately updated. Nothing about this sequence is spectacular — and that is precisely the point. The value arises not from dramatic finds but from the shift from “everything probably looks fine” to a checked “nothing unintended has changed since the last approved state”. How a single check run proceeds technically is shown in ScanU's step-by-step presentation.

Anyone wanting to get started does well with a plain sequence: first define a small set of important pages, then calm the test environment, then produce baselines and agree within the team who assesses and approves deviations — and only after that negotiate thresholds and check frequency. An English-language way into the topic is offered by ScanU's overview; the methodology behind it applies to every tool of this genre. Because that, in the end, is the central insight: the value of visual tests arises not in the algorithm that compares pixels but in the procedure around it — clean references, honest thresholds and people who assess differences instead of clicking them away.