Stabilize Screenshots for Visual Testing

Visual Test Options

TestingBot offers these features to stabilize your screenshots:

Ignore Specific Regions

You can ignore specific regions, defined in pixel coordinates, from the screenshot:

ignoreRegions = [{
  "x1": number,
  "y1": number,
  "x2": number,
  "y2": number
}]

Ignore Specific Selectors

Pass an array of CSS Selectors to ignore these DOM elements during the visual check.

ignoreSelectors = ["body div.test", "#sidebar"]

Only screenshot a specific selector

Pass a CSS selector to take a screenshot of this specific element only.

selector = "body div.test"

Scheduled Visual Tests

For visual tests you schedule in the dashboard, two more settings decide whether a run fails. Both are per test and need no code.

Mismatch threshold

A run fails when more than this share of pixels differs from the baseline. New tests start at 1%; you can set anything between 0.1% and 5% on the test's settings page. The slider shows how many of your last ten runs would have passed at the value you pick, so you can size it against your own history rather than guessing.

The comparison is tolerant of anti-aliasing before the threshold is applied: a pixel only counts as changed when a colour channel differs beyond a tolerance and at least one neighbouring pixel changed too. Sub-pixel font jitter and isolated stray pixels do not fail a run.

Ignored regions

Parts of a page that change on their own, a carousel, a clock, a rotating ad, will differ on every run and fail a test for no useful reason. On the test's settings page you can drag a box over those areas of the baseline to ignore them. Ignored regions are painted out of both the screenshot and the baseline before they are compared, so nothing inside one can fail a run. You can copy the regions you drew onto every browser in the test, scaled to each baseline's width.

After a failure, TestingBot also looks at the last three runs of that browser and proposes the regions that changed in every one of them. Nearby changes are merged into one rectangle, so a widget with several moving parts comes back as a single region rather than a box per line. These appear in amber on the settings page, drawn over the latest run, for you to accept or dismiss individually. Accepting one adds it to the ignored regions from the next run onwards. If most of the difference is the same on every run, the page itself has changed and you should approve a run as the new baseline instead.

The page of a failed run shows the same thing where you are already looking. Every region that changed is outlined and numbered on the screenshot: dashed amber for regions that changed on each of the last three runs, solid red for regions that are new in this run. The list next to it puts the new regions first, since those are the ones to check, and says how much of the page each one covers. You can ignore a repeating region from that list, or all of them at once from the summary at the top, without leaving the run. The summary also says what future runs would differ by once they are ignored, and whether that would pass. Ignoring regions changes future runs only; it does not change the result of the run you are looking at.

Regions are named after what the page says inside them, an element's label or the text in the region, so you can tell them apart without finding each box. A region with no text of its own is named by where it is on the page.

AI region names

With AI region names turned on, the changed regions of a failed run are also named and described, for example "Monthly price: went from $29 to $39" or "Navigation menu: Pricing link replaced with Enterprise". Up to eight regions are described per run, the largest first; a region that already has a name from an earlier run keeps it, and any others keep their usual name. These names carry an AI tag. They can be wrong, so check them against the image before you approve or reject a run. When the AI is unsure, the region keeps its usual name.

The feature is off until the account owner turns it on, either from the "What changed" list of a failed run or under AI region names for visual tests in Account Settings, where it can be turned off again at any time. It is separate from AI Insights, which sends test logs: turning on one does not turn on the other.

What is sent, after each failed run of a visual test, to our AI sub-processor Anthropic (United States):

• For up to eight changed regions, the largest first, a crop of the baseline and a crop of the run around the region, with a small margin, at most 512 pixels on the longest side. Not the full screenshots.
• The size and position of each region, and the text, labels and nearest heading the page shows in it. Secrets and personal data in that text are masked first, as for AI Insights.

The crops themselves are sent as they are: anything visible in them, such as a name or an email address shown on the page, is shared. Only turn this on if you are authorized to share screenshots of the pages you test, and consider ignoring regions that show personal data. Anthropic does not use data sent through its commercial API to train models and deletes inputs and outputs within 30 days. The names are stored with the run and removed with it. A region that repeats in the same place on later runs usually reuses its earlier name instead of being sent again. The descriptions count towards the same monthly AI limit per account as AI Insights.