Name: Visual Regression Testing Tutorial
Availability: InStock

In this chapter you write two visual tests, watch them create their baselines, break the page on purpose, read the diff, and then accept an intended change. That loop, run, fail, review, update, is all visual regression testing is. Everything later in the course makes it more reliable.

Set up a project#

You need Node.js 20 or newer. In an empty folder:

npm init -y
npm install -D @playwright/test@1.65.0
npx playwright install chromium

Pinning the exact version matters more for visual tests than for functional ones: a Playwright upgrade ships new browser builds, and new browser builds can render a few pixels differently. You want that to happen in a deliberate upgrade commit, not on a random Tuesday. If you already have a Playwright project, use it.

Create playwright.config.ts:

import { defineConfig, devices } from '@playwright/test';

export default defineConfig({
  testDir: './tests',
  fullyParallel: true,
  forbidOnly: !!process.env.CI,
  retries: process.env.CI ? 2 : 0,
  reporter: [['html', { open: 'never' }], ['list']],
  use: {
    trace: 'on-first-retry',
  },
  projects: [
    { name: 'chromium', use: { ...devices['Desktop Chrome'] } },
  ],
});

One browser for now. Chapter 5 adds more.

Two visual tests#

Create tests/todo.spec.ts:

import { test, expect } from '@playwright/test';

test.beforeEach(async ({ page }) => {
  await page.goto('https://demo.playwright.dev/todomvc/');
});

test('empty todo app', async ({ page }) => {
  await expect(page).toHaveScreenshot('todo-empty.png');
});

test('todo list with two items', async ({ page }) => {
  const input = page.getByPlaceholder('What needs to be done?');
  await input.fill('Buy milk');
  await input.press('Enter');
  await input.fill('Pay rent');
  await input.press('Enter');
  if (process.env.REGRESSION) {
    await page.addStyleTag({ content: '.todo-list li label { padding-left: 70px; }' });
  }
  await expect(page).toHaveScreenshot('todo-two-items.png');
});

The REGRESSION switch injects a CSS bug on demand so you can see a failure without editing the page. In a real suite the "bug" comes from your own code.

toHaveScreenshot() is a web-first assertion like toBeVisible(). It does more than take one picture:

It disables CSS animations and transitions, hides the text caret, and waits for web fonts to finish loading.
It takes screenshots until two consecutive ones are identical, so a page that is still settling does not get captured halfway.
It compares that stable screenshot with the baseline file.

If the screenshots never stabilise or never match within the assertion timeout (5 seconds by default), the assertion fails. The name argument is optional; without it Playwright generates one from the test title, but explicit names make the baseline files easier to find and review.

The first run fails, on purpose#

npx playwright test

Both tests fail:

Error: A snapshot doesn't exist at .../tests/todo.spec.ts-snapshots/todo-empty-chromium-darwin.png, writing actual.

There was nothing to compare against, so Playwright wrote the screenshots it took as the new baselines and failed the test anyway. That is deliberate: if a missing baseline passed silently, a CI run on a fresh checkout would "pass" without checking anything. The setting behind this is updateSnapshots, whose default ('default') is "create missing snapshots, but fail the tests that created them".

Look at the file names Playwright chose:

tests/todo.spec.ts-snapshots/
  todo-empty-chromium-darwin.png
  todo-two-items-chromium-darwin.png

The pattern is {name}-{projectName}-{platform}.png. chromium is the project from the config and darwin is the operating system that ran the test (it would be linux or win32 elsewhere). Playwright puts both in the name because the same page renders differently in different browsers and on different operating systems. Chapter 2 shows how big that difference is, and chapter 7 what it means for CI.

Open the baselines and check them before you commit them. A baseline is a statement that this picture is correct. If the page was already broken, you have just approved the bug.

Run again:

npx playwright test

  2 passed (671ms)

A new visual test fails on its very first run with "A snapshot doesn't exist ... writing actual". What should you do?

Set updateSnapshots: 'missing' in the config so this never fails again Open the PNG it wrote, confirm the page looks right, then commit it and re-run Delete the snapshot folder and run with --update-snapshots=all Raise the assertion timeout

The failure is expected: Playwright created the baseline and refuses to count that as a pass. Your job is to review the image before it becomes the reference. Making missing snapshots pass silently in the config is exactly what the default protects CI from.

Break it and read the diff#

Now introduce the regression:

REGRESSION=1 npx playwright test

  1) [chromium] › tests/todo.spec.ts:11:1 › todo list with two items

    Error: expect(page).toHaveScreenshot(expected) failed

      719 pixels (ratio 0.01 of all image pixels) are different.

      Snapshot: todo-two-items.png

    Expected: tests/todo.spec.ts-snapshots/todo-two-items-chromium-darwin.png
    Received: test-results/todo-todo-list-with-two-items-chromium/todo-two-items-actual.png
    Diff:     test-results/todo-todo-list-with-two-items-chromium/todo-two-items-diff.png

Your pixel count may differ slightly, since it depends on the fonts on your machine. The empty-app test still passes, because the regression only affects list items.

The failure gives you three images:

expected: the baseline.
actual: what the page looks like now.
diff: the actual image faded out, with every pixel that differs drawn in red. Ten pixels of padding on two labels shows up as two red text blocks.

The easiest way to compare them is the HTML report:

npx playwright show-report

Open the failed test. The image comparison has tabs for Diff, Actual and Expected, plus a side-by-side view and a slider that wipes from one image to the other. For layout shifts, the slider is the quickest way to see what moved.

The call log in the error is worth reading once, because it shows what the assertion did: disabled animations, waited for fonts, compared, waited 100 ms, took another screenshot, compared again. A real visual difference survives every retry; a page that was still loading usually does not.

A visual test failed on your machine and the diff image is mostly red text in one area. What is the quickest way to tell whether the text moved or changed?

Open the HTML report and use the slider or side-by-side view between Expected and Actual Re-run the test with --update-snapshots and look at the new file Count the red pixels in the diff

The diff only says which pixels differ. Comparing expected and actual directly shows whether something shifted, changed colour or changed content. Updating the snapshot first would throw away the evidence you need to make that call.

Accepting an intended change#

Suppose the extra padding was a deliberate design change. You do not edit PNG files by hand; you let Playwright rewrite the baseline:

REGRESSION=1 npx playwright test --update-snapshots=changed

.../todo-two-items-chromium-darwin.png is re-generated, writing actual.
  2 passed

The modes of --update-snapshots (short form -u) are:

Mode	What it rewrites
`changed`	Only baselines that did not match, plus missing ones. The default when you pass the flag without a value.
`missing`	Only baselines that do not exist yet. Tests that create them pass.
`all`	Every baseline of every test that ran, even ones that matched.
`none`	Nothing.

Use changed for normal work. all rewrites files that were fine, which creates noisy commits with tiny invisible differences. Narrow the run to what you mean to change, with a file, --grep or --project, so an update for one page does not quietly re-approve a broken one.

Then treat the baseline like code:

git diff --stat tests/
git add tests/todo.spec.ts-snapshots/todo-two-items-chromium-darwin.png
git commit -m "Wider todo label padding: update baseline"

On GitHub and most code review tools, a changed PNG in a pull request shows the old and new images side by side, so the reviewer approves the visual change together with the CSS that caused it. That pairing is the review workflow; chapter 7 builds on it.

What to commit#

Commit the *-snapshots/ folders: they are the expected results, as much part of the test as the assertions. Do not commit test-results/ or playwright-report/; Playwright's generated .gitignore already excludes them. PNG baselines are usually a few tens of kilobytes each, so even a few hundred are fine in git. If you collect thousands, consider Git LFS for the snapshot folders.

What you have now#

Two tests, two committed baselines, and the loop: a change fails the test, the report shows you the difference, and you either fix the code or update the baseline on purpose. The next chapter deals with the biggest practical problem of visual testing: screenshots that differ when nothing is wrong.