Visual Testing on a Cloud Grid with TestingBot

Chapter 5 ended with a gap: Playwright's bundled engines on Linux are not Chrome on Windows or Safari on a Mac, and visual tests care about exactly that difference. This chapter closes it in two ways. First, the toHaveScreenshot() suite you already have runs unchanged on remote browsers on the TestingBot grid. Second, TestingBot has its own visual comparison, which keeps baselines on the server and gives you a review screen instead of PNG files in git. Both are shown with runs we did on the grid while writing this chapter.

Option 1: your Playwright suite on remote browsers#

Playwright can connect to a browser running elsewhere instead of launching one locally. Everything else, including toHaveScreenshot(), stays the same: the remote browser takes the screenshot, Playwright transfers it back and compares it with the baseline on your disk. Baselines stay in git, and the review workflow from chapters 1 and 7 does not change.

Create testingbot.ts with a helper that builds the endpoint URL:

type TestingBotBrowser = {
  browserName: 'chrome' | 'edge' | 'firefox' | 'safari';
  browserVersion?: string;
  platform?: string;
};

export function testingbotEndpoint(browser: TestingBotBrowser): string {
  const capabilities = {
    browserVersion: 'latest',
    ...browser,
    'tb:options': {
      key: process.env.TB_KEY,
      secret: process.env.TB_SECRET,
      build: process.env.TB_BUILD ?? 'visual-local',
    },
  };
  return `wss://cloud.testingbot.com/playwright?capabilities=${encodeURIComponent(JSON.stringify(capabilities))}`;
}

Then a separate config, playwright.testingbot.config.ts:

import { defineConfig } from '@playwright/test';
import baseConfig from './playwright.config';
import { testingbotEndpoint } from './testingbot';

export default defineConfig({
  ...baseConfig,
  workers: 3,
  // One baseline folder per grid project. No {platform}: that token is the OS
  // running the test runner, not the OS of the remote browser.
  snapshotPathTemplate: '{testDir}/__screenshots__/{projectName}/{testFilePath}/{arg}{ext}',
  use: { ...baseConfig.use, viewport: { width: 1280, height: 720 } },
  projects: [
    {
      name: 'chrome-win11',
      use: {
        browserName: 'chromium',
        connectOptions: { wsEndpoint: testingbotEndpoint({ browserName: 'chrome', platform: 'WIN11' }) },
      },
    },
    {
      name: 'firefox-win11',
      use: {
        browserName: 'firefox',
        connectOptions: { wsEndpoint: testingbotEndpoint({ browserName: 'firefox', platform: 'WIN11' }) },
      },
    },
    {
      name: 'safari-sequoia',
      use: {
        browserName: 'webkit',
        connectOptions: { wsEndpoint: testingbotEndpoint({ browserName: 'safari', platform: 'SEQUOIA' }) },
      },
    },
  ],
});

The snapshotPathTemplate deserves a closer look. Playwright's default file name ends in {platform}, which is process.platform of the machine running the test runner. With remote browsers, that is the wrong OS: a laptop and a Linux CI job would write -darwin and -linux files for the same remote Windows browser. Naming the folder after the project, which already says browser and OS, gives one baseline per remote configuration no matter where the runner is.

The project's browserName must match the remote browser's engine: chromium for Chrome and Edge, firefox for Firefox, webkit for Safari. The Playwright capabilities generator lists every browser and platform value.

Reporting pass or fail to the dashboard#

TestingBot records a video and logs for every session but cannot know whether your assertions passed. An automatic fixture reports it. Put this in tests/fixtures.ts and import test and expect from it in the grid specs:

import { test as base, expect, type Page } from '@playwright/test';

function executor(action: string, args: Record<string, unknown>): string {
  return `testingbot_executor: ${JSON.stringify({ action, arguments: args })}`;
}

export const test = base.extend<{ testingbotStatus: void }>({
  testingbotStatus: [async ({ page }, use, testInfo) => {
    const remote = !!testInfo.project.use.connectOptions;
    if (remote) {
      await page.evaluate(() => {}, executor('setSessionName', { name: testInfo.title }));
    }
    await use();
    if (remote) {
      await page.evaluate(() => {}, executor('setSessionStatus', {
        passed: testInfo.status === testInfo.expectedStatus,
        reason: testInfo.error?.message?.slice(0, 200) ?? '',
      }));
    }
  }, { auto: true }],
});

export { expect };

A visual test file for the grid, tests/grid/todo.visual.spec.ts:

import { test, expect } from '../fixtures';

test.beforeEach(async ({ page }) => {
  await page.goto('https://demo.playwright.dev/todomvc/');
});

test('empty todo app', async ({ page }) => {
  await expect(page).toHaveScreenshot('todo-empty.png');
});

test('todo list with two items', async ({ page }) => {
  const input = page.getByPlaceholder('What needs to be done?');
  for (const item of ['Buy milk', 'Pay rent']) {
    await input.fill(item);
    await input.press('Enter');
  }
  await expect(page.locator('.todoapp')).toHaveScreenshot('todo-two-items.png');
});

Create the baselines, review them, then run again:

export TB_KEY=your-key TB_SECRET=your-secret TB_BUILD=visual-grid
npx playwright test -c playwright.testingbot.config.ts --update-snapshots=missing
npx playwright test -c playwright.testingbot.config.ts

We ran this in Chrome and Firefox on Windows 11 and Safari on macOS Sequoia: six baselines in tests/__screenshots__/<project>/, and on the next runs all six tests passed and showed as passed sessions in the build in the TestingBot dashboard. The screenshot is rendered on the remote machine, so with the path template above it does not matter whether the runner is your laptop or a CI job: both compare against the same baseline for that browser and OS.

How many sessions run at once depends on your plan; extra connections wait in a queue. More on the setup in our Playwright Test integration guide.

Option 2: TestingBot's visual comparison#

TestingBot can also take and compare the screenshot itself. You send a command with a snapshot name; the first time, TestingBot stores the screenshot as the baseline, and every later call with that name is compared with it on the server. The result comes back to your test within seconds, and each comparison appears in the member area with the baseline, the new screenshot and a diff image. Baselines live on TestingBot rather than in your repository, and approving a change is a click instead of a commit.

From Playwright, the command is a page.evaluate call with a visual.snapshot action. Add a helper to tests/fixtures.ts:

type VisualResult = { match: boolean; pixelDifference?: number; url?: string; error?: string };

// TestingBot's server-side visual comparison (Chromium-based browsers over Playwright).
// The project name goes into the snapshot name so each browser keeps its own baseline.
export async function tbSnapshot(page: Page, name: string, options: Record<string, unknown> = {}) {
  const fullName = `${name}-${test.info().project.name}`;
  return (await page.evaluate(() => {}, executor('visual.snapshot', { name: fullName, options }))) as VisualResult;
}

and use it in a test:

import { test, expect, tbSnapshot } from '../fixtures';

// visual.snapshot over Playwright needs a Chromium-based browser (Chrome or Edge).
test.skip(({ browserName }) => browserName !== 'chromium', 'Chromium-based browsers only');

test('todo app, compared by TestingBot', async ({ page }) => {
  await page.goto('https://demo.playwright.dev/todomvc/');
  const input = page.getByPlaceholder('What needs to be done?');
  for (const item of ['Buy milk', 'Pay rent']) {
    await input.fill(item);
    await input.press('Enter');
  }

  const result = await tbSnapshot(page, 'todomvc-two-items', {
    ignoreSelectors: ['.info'],
  });
  expect(result.error, result.error).toBeUndefined();
  expect(result.match, `Visual difference, review it at ${result.url}`).toBe(true);
});

The first run returned { match: true, pixelDifference: 0 } and created the baseline. Later runs return the comparison with links:

{ match: false, pixelDifference: 874, visualId: ..., runId: ...,
  url: 'https://testingbot.com/members/visual/<visualId>/runs/<runId>' }

That failing result came from injecting the padding regression from chapter 1; the assertion message carries the link straight to the review page.

Two things we learned running this:

• Over Playwright, it works with Chrome and Edge only. In Firefox and Safari Playwright sessions the command returned match: false with an error saying it could not capture a screenshot, which is why the expect(result.error) line is there and why the test skips other engines. For Firefox and Safari, use the Selenium form below.
• Give every browser configuration its own snapshot name. Put the browser and OS (here, the project name) into the name. When we reused one name across Firefox and Safari, Firefox was compared with Safari's screenshot and failed with about 19,600 different pixels on an unchanged page. With per-browser names, every repeat run matched.

The options include threshold (per-pixel colour tolerance from 0 to 1, default 0.1), antialiasing (default on: anti-aliased pixels are not counted), ignoreRegions (rectangles in pixel coordinates), ignoreSelectors (CSS selectors to leave out) and selector (screenshot one element only). To replace a baseline from code, send visual.baseline with the same name. The Playwright visual testing docs list the options and the response fields.

From Selenium, including Firefox and Safari#

Selenium sends the same command through executeScript, and this works in Firefox and Safari too. This Node.js script ran green, repeatedly, in Firefox on Windows 11 and Safari on macOS Sequoia:

// node selenium-visual.mjs firefox WIN11   |   node selenium-visual.mjs safari SEQUOIA
import { Builder } from 'selenium-webdriver';

const [browserName = 'firefox', platformName = 'WIN11'] = process.argv.slice(2);
const { TB_KEY, TB_SECRET } = process.env;

const driver = await new Builder()
  .usingServer(`https://${TB_KEY}:${TB_SECRET}@hub.testingbot.com/wd/hub`)
  .withCapabilities({
    browserName,
    browserVersion: 'latest',
    platformName,
    'tb:options': { name: 'todo app visual check', build: process.env.TB_BUILD ?? 'visual-local' },
  })
  .build();

let passed = false;
try {
  await driver.get('https://demo.playwright.dev/todomvc/');
  // Put the browser and OS in the name: each configuration needs its own baseline.
  const name = `todomvc-home-${browserName}-${platformName}`.toLowerCase();
  const command = JSON.stringify({ name, options: { threshold: 0.1 } });
  const result = await driver.executeScript(`tb:visual.snapshot=${command}`);
  console.log(browserName, result);
  passed = result.match === true;
} finally {
  await driver.executeScript(`tb:test-result=${passed ? 'passed' : 'failed'}`);
  await driver.quit();
}
process.exitCode = passed ? 0 : 1;

The Selenium visual testing docs show the command in Java, Python, C#, Ruby and PHP, plus a fullpage option for Chrome, Edge and Firefox. There are equivalent pages for Appium and Puppeteer.

Reviewing changes in the dashboard#

Each comparison is a run under Visual Testing in the member area. A failed run shows the baseline, the new screenshot and a diff image highlighting the changed pixels, as described on visual testing results. From the run page you approve or reject it, and approving can make the new screenshot the baseline for future runs. A review queue lists the failed runs nobody has approved or rejected yet, so changes do not pile up unseen.

This is the main trade-off against Option 1. With toHaveScreenshot(), baselines are files reviewed in pull requests, versioned with the code. With TestingBot's comparison, baselines are shared server-side state reviewed in a dashboard, which suits teams where designers or product owners approve changes without touching git.

Option 3: scheduled visual tests without code#

For pages you want to watch rather than test on every commit, such as marketing pages, documentation or a production checkout, TestingBot runs codeless visual tests: enter a URL, choose up to five browser configurations (each with a screen resolution), and a schedule. When a run differs from its baseline, you get an email or SMS alert with the baseline, screenshot and diff.

Scheduled tests have the settings this course has been building towards, without code:

• A mismatch threshold per test, 1% by default and adjustable between 0.1% and 5%, with a preview of how your last ten runs would have fared at the value you pick.
• Ignored regions you draw over the baseline, and suggested regions for areas that changed in each of the last three runs.
• A comparison that tolerates anti-aliasing: a pixel only counts when it differs beyond a tolerance and a neighbouring pixel changed too.
• Automatic stabilisation before each screenshot: CSS animations frozen, lazy images loaded for full-page captures, the page's date and time frozen, the caret and scrollbars hidden, and on mobile devices the status bar removed. The stabilisation guide has the details.

A scheduled test can also run on demand from CI against a pull request's preview deployment, reporting back as a GitHub check, which the next chapter covers.

Which option to choose#

Many teams combine them: component and page tests with toHaveScreenshot() in pull requests, and a scheduled test watching the production homepage and checkout.