Run Visual Tests from CI

Allow preview hosts

The preview URL replaces the test's URL for that one run. It may be on the same domain as the test's URL (pr-42.staging.example.com for a test of www.example.com). Preview deployments on another domain, such as *.vercel.app, must be listed under Preview hosts in the test's settings. A * matches within one part of the host name.

The preview URL must resolve to a public address. To check a deployment that is only reachable inside your network, start a TestingBot Tunnel first: hosted visual tests use a running tunnel automatically.

GitHub Actions example

Install the TestingBot GitHub App on the repository owner so TestingBot can post the check. Store your API key and secret as the repository secrets TESTINGBOT_KEY and TESTINGBOT_SECRET, then add a job after the one that deploys the preview. Replace 1234 with the ID of your visual test (it is in the test's URL in the dashboard).

name: Visual check
on: pull_request

jobs:
  deploy:
    runs-on: ubuntu-latest
    outputs:
      preview_url: ${{ steps.deploy.outputs.url }}
    steps:
      # Your existing preview deployment, exposing the preview URL as an output.
      - id: deploy
        run: echo "url=https://pr-${{ github.event.number }}.preview.example.com/" >> "$GITHUB_OUTPUT"

  visual:
    needs: deploy
    runs-on: ubuntu-latest
    steps:
      - name: Run the visual test against the preview
        env:
          TB_KEY: ${{ secrets.TESTINGBOT_KEY }}
          TB_SECRET: ${{ secrets.TESTINGBOT_SECRET }}
          PREVIEW_URL: ${{ needs.deploy.outputs.preview_url }}
          HEAD_SHA: ${{ github.event.pull_request.head.sha }}
          PR: ${{ github.event.pull_request.number }}
        run: |
          curl -sSf -u "$TB_KEY:$TB_SECRET" -X POST \
            "https://api.testingbot.com/v1/visual/tests/1234/run" \
            --data-urlencode "url=$PREVIEW_URL" \
            -d "repo=$GITHUB_REPOSITORY" \
            -d "commit=$HEAD_SHA" \
            -d "pr=$PR"

The call returns as soon as the browsers are queued. The TestingBot / visual #1234 check shows as in progress and concludes when the last browser has been compared. Use the head.sha of the pull request, not the merge commit, so the check lands on the commit branch protection looks at. To block merging on a visual change, add the check under Require status checks to pass in the branch protection rule.

Poll instead of a check

Leave out repo and commit when you do not use the GitHub App, and let the job wait for the result itself:

CHECK=$(curl -sSf -u "$TB_KEY:$TB_SECRET" -X POST \
  "https://api.testingbot.com/v1/visual/tests/1234/run" \
  --data-urlencode "url=$PREVIEW_URL" | jq -r .check)

for i in $(seq 1 90); do
  STATUS=$(curl -sSf -u "$TB_KEY:$TB_SECRET" "https://api.testingbot.com/v1/visual/checks/$CHECK")
  if [ "$(echo "$STATUS" | jq -r .state)" = "done" ]; then
    echo "$STATUS" | jq '.browsers[] | {browser, state, passed, mismatch, run_url}'
    [ "$(echo "$STATUS" | jq -r .passed)" = "true" ]
    exit $?
  fi
  sleep 20
done
echo "Visual check did not finish in time"
exit 1

How pass/fail is decided

Each browser of the test is compared with its own baseline, using the test's mismatch threshold, masks and match levels, exactly as a scheduled run would. The check passes only when every browser was compared and passed. It fails when:

• a browser's screenshot differs from its baseline by more than the threshold;
• a browser has no baseline yet (see below);
• a browser could not be run, for example because the preview did not load.

The check lists every browser with its mismatch percentage and a link to the run page, where you can review the difference and approve it.

Baselines

A run against a preview URL is compared with the test's normal baselines and never replaces them on its own, so scheduled runs of the test are not affected. A preview run also does not send the test's alerts: its result is reported on the check.

A browser that has no baseline yet is not compared. Run the test once against its own URL, on its schedule or with Run now, to record one. When a change is intended, approve the run on its run page and set it as the new baseline, the same way as for a scheduled run.