Thresholds: How Much Difference to Allow

Once screenshots are stable, the remaining question is how strict the comparison should be. Playwright has three knobs for it, and they are easy to misread: one decides whether a single pixel counts as different, the other two decide how many different pixels a test may have. This chapter measures what each one does on a real element, so you can choose values instead of guessing.

How Playwright compares two images#

The default comparator walks both images pixel by pixel. For each pair it computes a colour difference in the YIQ colour space, which weights brightness more than hue, roughly the way human eyes do. If the difference is above threshold, the pixel counts as different. Then it counts the different pixels and compares the count with maxDiffPixels or maxDiffPixelRatio.

So with the defaults, a test fails as soon as one pixel differs by more than 0.2.

An experiment with a button#

Create tests/threshold.spec.ts. It renders a button in one of four variants, chosen by an environment variable, and takes the comparison options from another:

import { test, expect } from '@playwright/test';

const variant = process.env.VARIANT ?? 'base';
const styles: Record<string, string> = {
  base:    'background:#1a73e8; padding:12px 24px',
  nudge:   'background:#1b74e9; padding:12px 24px',          // +1 on every colour channel
  rebrand: 'background:#1e88e5; padding:12px 24px',          // a visibly different blue
  shift:   'background:#1a73e8; padding:12px 23px 12px 25px', // text 1px to the right
};

test('buy button', async ({ page }) => {
  await page.setContent(`
    <button style="${styles[variant]}; color:white; border:0;
                   font:600 16px system-ui; border-radius:6px">Buy now</button>`);
  const options = JSON.parse(process.env.OPTS ?? '{}');
  await expect(page.getByRole('button')).toHaveScreenshot('buy-button.png', options);
});

Record the baseline with the base variant, then run each variant against it:

npx playwright test tests/threshold.spec.ts --update-snapshots=missing
VARIANT=rebrand npx playwright test tests/threshold.spec.ts
VARIANT=rebrand OPTS='{"threshold":0.05}' npx playwright test tests/threshold.spec.ts

These are the results we got (Chromium on macOS; your counts will be close but not identical):

Three lessons are in that table.

The default threshold lets real colour changes through. Changing the brand blue from #1a73e8 to #1e88e5 is a change a designer would notice and might want reviewed, and with the defaults it passes. Every pixel of the button changed, but each by less than 0.2 in YIQ terms. If colours matter to you (brand colours, status colours, contrast fixes), lower threshold to around 0.05 to 0.1.

threshold: 0 is too strict for real use. A change of one unit on each colour channel, invisible to anyone, fails 89% of the button. Rendering noise between runs, GPU paths and browser updates produce exactly this kind of tiny change.

Pixel allowances and colour tolerance cover different failures. A one-pixel text shift changes few pixels, but changes them a lot (white text becomes blue background), so threshold does not help and maxDiffPixels does. A colour change touches many pixels by a little, so maxDiffPixels does not hide it and threshold does. Tune the knob that matches the noise you are actually seeing.

Why a big allowance is dangerous#

maxDiffPixels: 150 made the one-pixel shift pass. It would also make many real bugs pass: a missing icon, a changed digit in a price, a one-word label change. On a full-page screenshot of 1280x720 (921,600 pixels), maxDiffPixelRatio: 0.01 allows 9,216 different pixels, enough for a whole paragraph to vanish.

That is why stabilisation comes first (chapter 2) and tolerance second. A useful process:

1. Make the test stable with the defaults if you can. Many tests need no tolerance at all, because the same browser in the same environment produces identical pixels.
2. If a test still fails intermittently, look at the diff image. Speckles along text edges point to anti-aliasing noise; a block of red points to content you did not stabilise.
3. For edge noise, add the smallest maxDiffPixels that covers it, on that one test, and keep it well below the size of anything you care about.
4. Write down why. A comment next to the option saves the next person from doubling it.

Prefer maxDiffPixels for element and component screenshots, where the image size is small and known, and maxDiffPixelRatio only where image size varies, such as full-page screenshots of pages with dynamic length.

Setting defaults for the whole suite#

Per-test options override suite defaults, which live in the config:

export default defineConfig({
  expect: {
    toHaveScreenshot: {
      threshold: 0.1,
      maxDiffPixels: 0,
      animations: 'disabled',
      caret: 'hide',
    },
  },
});

Keep the global values strict and loosen individual tests, not the other way round. A loose global default silently weakens every new test somebody adds later.

Pixel comparison and "perceptual" comparison#

Playwright's comparator is a pixel comparison with a perceptual colour metric: YIQ distances, plus detection of anti-aliased pixels along edges. Other tools describe their algorithms differently, but most fall into the same few ideas:

• Exact pixel equality. Fast and simple, and fails on any rendering noise. Usable only in a perfectly controlled environment.
• Per-pixel colour tolerance (Playwright's threshold, and the threshold option of TestingBot's visual comparison in chapter 6). Ignores small colour shifts.
• Anti-aliasing detection. Ignores pixels that differ only because an edge was smoothed differently. TestingBot's comparison does this by default (its antialiasing option), and its scheduled visual tests additionally only count a pixel when a neighbouring pixel changed too, so isolated stray pixels do not fail a run.
• Structural or AI-based comparison. Some commercial tools compare layout and content structure rather than raw pixels, to ignore noise and group related changes. These trade transparency for fewer false failures; you can no longer reason about exactly which pixel failed.

Whatever the tool, the decision is the same one you made above: which differences are noise in your environment, and which ones must a human see.

Where the threshold is not the answer#

If you find yourself raising tolerances again and again for one test, the test is probably wrong, not the threshold:

• The screenshot covers too much. A full page with a news feed will never be stable; screenshot the component you care about (chapter 4).
• Something dynamic is not stabilised. Go back to the chapter 2 checklist.
• The baseline came from a different environment. A macOS baseline compared on Linux is thousands of pixels off, and no sane tolerance fixes that.

The next chapter is about choosing what to screenshot: the whole page, one element, or one component at several screen sizes.