Visual Regression Testing in CI with GitHub Actions

A visual suite earns its keep in CI: every pull request gets compared with the approved baselines before it merges, and a reviewer sees the visual change next to the code that caused it. This chapter builds that workflow with GitHub Actions. The hard parts are not the YAML; they are where baselines come from and how a pull request updates them.

The baseline problem in CI#

Chapter 2 showed that a macOS baseline fails on Linux by thousands of pixels. GitHub's ubuntu-latest runners are Linux. So you need Linux baselines, created in an environment identical to CI. There are three ways to get them:

1. Run CI in the Playwright Docker image, and create baselines in the same image on your machine. Same OS, same fonts, same browser builds.
2. Let CI create the baselines in a separate, manually triggered job that commits them.
3. Run the browsers remotely (chapter 6), so the runner's OS does not matter at all.

Most teams combine 1 and 2: developers can update locally with Docker, and anyone can update from the pull request without installing anything.

The test workflow#

Create .github/workflows/visual.yml:

name: Visual tests
on:
  pull_request:
  push:
    branches: [ main ]

jobs:
  visual:
    runs-on: ubuntu-latest
    container:
      image: mcr.microsoft.com/playwright:v1.65.0-noble
      options: --user 1001
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version: lts/*
      - run: npm ci
      - name: Run visual tests
        run: npx playwright test
      - uses: actions/upload-artifact@v4
        if: ${{ !cancelled() }}
        with:
          name: visual-report
          path: |
            playwright-report/
            test-results/
          retention-days: 14

The details that matter for visual tests:

• The container image tag matches @playwright/test exactly. The image contains the browser builds for that version plus the fonts and libraries they render with. When you upgrade Playwright, change the tag in the same commit and expect to update baselines (chapter 5).
• The same image is what developers use to create baselines locally (below). That is the whole trick: baselines and CI see identical pixels.
• Upload test-results/ as well as the report. The -actual.png, -expected.png and -diff.png files of every failed comparison are in test-results/, and the HTML report links to them. When a pull request fails, the reviewer downloads visual-report, runs npx playwright show-report playwright-report and uses the slider view from chapter 1.
• if: ${{ !cancelled() }} uploads the report when tests fail, which is the only time you need it.
• Leave updateSnapshots at its default. A missing baseline fails the job instead of being silently created on a throwaway runner.

The snapshot files from the container end in -linux.png. Commit those; -darwin.png files from a Mac are not used in CI and only cause confusion, so many teams add a rule to delete or .gitignore them.

Creating Linux baselines on your machine#

With Docker installed, run the suite inside the same image CI uses:

docker run --rm -v "$PWD":/work -w /work --ipc=host \
  mcr.microsoft.com/playwright:v1.65.0-noble \
  npx playwright test --update-snapshots=changed

This writes -linux.png baselines into your working copy. Run it once without --update-snapshots afterwards: when we did this for the course tests, the Linux baselines passed in the container every time, while the macOS ones had failed there by 1,959 to 3,541 pixels per test.

If node_modules was installed on macOS, run npm ci inside the container first (or mount a separate node_modules), because some packages ship OS-specific binaries. @playwright/test itself is pure JavaScript, which is why the command above works with a Mac node_modules.

Updating baselines from a pull request#

Asking every contributor to run Docker is a barrier. A second workflow lets anyone update the baselines of a pull request from the GitHub UI. Create .github/workflows/update-baselines.yml:

name: Update visual baselines
on:
  workflow_dispatch:

permissions:
  contents: write

jobs:
  update:
    runs-on: ubuntu-latest
    container:
      image: mcr.microsoft.com/playwright:v1.65.0-noble
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version: lts/*
      - run: npm ci
      - name: Update changed and missing baselines
        run: npx playwright test --update-snapshots=changed
      - name: Commit new baselines
        run: |
          git config --global --add safe.directory "$GITHUB_WORKSPACE"
          git config user.name "github-actions[bot]"
          git config user.email "41898282+github-actions[bot]@users.noreply.github.com"
          git add -A '*-snapshots/*.png'
          git diff --cached --quiet || git commit -m "Update visual baselines"
          git push

To use it: open Actions, pick Update visual baselines, choose the pull request's branch and run it. The job checks out that branch, regenerates only the baselines that differ, and pushes a commit to the branch. The pull request then shows the changed PNGs, and GitHub's image diff (2-up, swipe and onion skin views) lets the reviewer compare old and new.

Points to get right:

• Manual trigger only. Never update baselines automatically on every push; a regression would approve itself.
• --update-snapshots=changed, not all, so unchanged baselines are not rewritten with invisible noise.
• The approval is the code review. The pull request now contains the code change and the image change together. Whoever approves the pull request approves the new look. Branch protection that requires a review covers baselines automatically.
• safe.directory is needed because the container runs as root while the checkout belongs to another user; without it git refuses to work in the folder.
• The default GITHUB_TOKEN cannot push to branches of forks, and a push made with it does not trigger other workflows. Run the visual workflow again manually after the update, or use a token from a GitHub App if you need it to start automatically.

Running the visual suite on the TestingBot grid from CI#

If your visual tests use remote browsers (chapter 6), CI needs no browsers and no container. Store your key and secret as repository secrets and run the grid config:

    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version: lts/*
      - run: npm ci
      - name: Visual tests on TestingBot
        run: npx playwright test -c playwright.testingbot.config.ts
        env:
          TB_KEY: ${{ secrets.TB_KEY }}
          TB_SECRET: ${{ secrets.TB_SECRET }}
          TB_BUILD: visual-${{ github.run_id }}

Because the baselines are named per remote project (chapter 6), they match whether the runner is a laptop or this Linux job. We checked this by running the grid suite from the Linux Playwright container against baselines recorded from a Mac: all six comparisons passed. The update workflow above works the same way, with -c playwright.testingbot.config.ts added to its test command.

If the app under test runs inside the CI job (for example on localhost:3000), the remote browsers need TestingBot Tunnel to reach it; the GitHub Actions integration guide shows the tunnel action and the secrets setup.

Scheduled TestingBot visual tests as a pull request check#

TestingBot's scheduled visual tests (chapter 6) can also run from CI. Your workflow deploys a preview of the pull request, then calls the TestingBot API to run an existing visual test against the preview URL in all of the test's browsers. With the TestingBot GitHub App installed, the result appears as a check on the pull request's commit, which you can make required in branch protection. Without the app, the job can poll for the result instead. A preview run compares with the test's normal baselines and never replaces them; when the change is intended, you approve it on the run page. The visual tests in CI guide has the full workflow.

Keeping the suite fast#

Visual suites grow quickly, and slow pull request checks get skipped. Three habits keep them fast:

• Run all browsers and breakpoints nightly, and a single browser on pull requests (--project=chromium).
• Prefer element screenshots (chapter 4): they are faster to capture and compare than full pages.
• Shard large suites across jobs with --shard=1/4 and merge the reports, exactly as for functional tests. The Playwright course chapter on CI shows the matrix and merge job.