--- title: Visual Testing on a Cloud Grid with TestingBot | TestingBot description: 'Chapter 5 ended with a gap: Playwright's bundled engines on Linux are not Chrome on Windows or Safari on a Mac, and visual tests care about exactly that…' source_url: html: https://testingbot.com/resources/courses/visual-regression-testing/visual-testing-on-a-cloud-grid md: https://testingbot.com/resources/courses/visual-regression-testing/visual-testing-on-a-cloud-grid.md --- Course [Visual Regression Testing Tutorial](https://testingbot.com/resources/courses/visual-regression-testing) Chapter 7 of 10 1. [1 Visual Regression Testing Tutorial](https://testingbot.com/resources/courses/visual-regression-testing) 2. [2 Your First Screenshot Comparison with Playwright](https://testingbot.com/resources/courses/visual-regression-testing/first-screenshot-comparison) 3. [3 Making Visual Tests Stable](https://testingbot.com/resources/courses/visual-regression-testing/stable-screenshots) 4. [4 Thresholds: How Much Difference to Allow](https://testingbot.com/resources/courses/visual-regression-testing/thresholds-and-diff-tolerance) 5. [5 Full Page, Element and Component Snapshots](https://testingbot.com/resources/courses/visual-regression-testing/page-element-and-component-snapshots) 6. [6 Cross-Browser Visual Testing](https://testingbot.com/resources/courses/visual-regression-testing/cross-browser-visual-testing) 7. [7 Visual Testing on a Cloud Grid with TestingBot](https://testingbot.com/resources/courses/visual-regression-testing/visual-testing-on-a-cloud-grid) 8. [8 Visual Regression Testing in CI with GitHub Actions](https://testingbot.com/resources/courses/visual-regression-testing/visual-tests-in-github-actions) 9. [9 A Visual Testing Strategy That Lasts](https://testingbot.com/resources/courses/visual-regression-testing/visual-testing-strategy) 10. [10 Visual Regression Testing Quiz](https://testingbot.com/resources/courses/visual-regression-testing/knowledge-check) [Exit course](https://testingbot.com/resources/courses) Chapter 7 of 10 Intermediate 12 min read 2 quick checks Updated Nov 2026 # Visual Testing on a Cloud Grid with TestingBot On this page 1. [Option 1: your Playwright suite on remote browsers](https://testingbot.com#option-1-your-playwright-suite-on-remote-browsers) 2. [Reporting pass or fail to the dashboard](https://testingbot.com#reporting-pass-or-fail-to-the-dashboard) 3. [Option 2: TestingBot's visual comparison](https://testingbot.com#option-2-testingbot-s-visual-comparison) 4. [Reviewing changes in the dashboard](https://testingbot.com#reviewing-changes-in-the-dashboard) 5. [Option 3: scheduled visual tests without code](https://testingbot.com#option-3-scheduled-visual-tests-without-code) 6. [Which option to choose](https://testingbot.com#which-option-to-choose) Chapter 5 ended with a gap: Playwright's bundled engines on Linux are not Chrome on Windows or Safari on a Mac, and visual tests care about exactly that difference. This chapter closes it in two ways. First, the `toHaveScreenshot()` suite you already have runs unchanged on remote browsers on the TestingBot grid. Second, TestingBot has its own visual comparison, which keeps baselines on the server and gives you a review screen instead of PNG files in git. Both are shown with runs we did on the grid while writing this chapter. ## Option 1: your Playwright suite on remote browsers[#](https://testingbot.com#option-1-your-playwright-suite-on-remote-browsers) Playwright can connect to a browser running elsewhere instead of launching one locally. Everything else, including `toHaveScreenshot()`, stays the same: the remote browser takes the screenshot, Playwright transfers it back and compares it with the baseline on your disk. Baselines stay in git, and the review workflow from chapters 1 and 7 does not change. Create `testingbot.ts` with a helper that builds the endpoint URL: ```typescript type TestingBotBrowser = { browserName: 'chrome' | 'edge' | 'firefox' | 'safari'; browserVersion?: string; platform?: string; }; export function testingbotEndpoint(browser: TestingBotBrowser): string { const capabilities = { browserVersion: 'latest', ...browser, 'tb:options': { key: process.env.TB_KEY, secret: process.env.TB_SECRET, build: process.env.TB_BUILD ?? 'visual-local', }, }; return `wss://cloud.testingbot.com/playwright?capabilities=${encodeURIComponent(JSON.stringify(capabilities))}`; } ``` Then a separate config, `playwright.testingbot.config.ts`: ```typescript import { defineConfig } from '@playwright/test'; import baseConfig from './playwright.config'; import { testingbotEndpoint } from './testingbot'; export default defineConfig({ ...baseConfig, workers: 3, // One baseline folder per grid project. No {platform}: that token is the OS // running the test runner, not the OS of the remote browser. snapshotPathTemplate: '{testDir}/ __screenshots__ /{projectName}/{testFilePath}/{arg}{ext}', use: { ...baseConfig.use, viewport: { width: 1280, height: 720 } }, projects: [ { name: 'chrome-win11', use: { browserName: 'chromium', connectOptions: { wsEndpoint: testingbotEndpoint({ browserName: 'chrome', platform: 'WIN11' }) }, }, }, { name: 'firefox-win11', use: { browserName: 'firefox', connectOptions: { wsEndpoint: testingbotEndpoint({ browserName: 'firefox', platform: 'WIN11' }) }, }, }, { name: 'safari-sequoia', use: { browserName: 'webkit', connectOptions: { wsEndpoint: testingbotEndpoint({ browserName: 'safari', platform: 'SEQUOIA' }) }, }, }, ], }); ``` The `snapshotPathTemplate` deserves a closer look. Playwright's default file name ends in `{platform}`, which is `process.platform` of the machine running the test runner. With remote browsers, that is the wrong OS: a laptop and a Linux CI job would write `-darwin` and `-linux` files for the same remote Windows browser. Naming the folder after the project, which already says browser and OS, gives one baseline per remote configuration no matter where the runner is. The project's `browserName` must match the remote browser's engine: `chromium` for Chrome and Edge, `firefox` for Firefox, `webkit` for Safari. The [Playwright capabilities generator](https://testingbot.com/support/web-automate/playwright/capabilities) lists every browser and platform value. ## Reporting pass or fail to the dashboard[#](https://testingbot.com#reporting-pass-or-fail-to-the-dashboard) TestingBot records a video and logs for every session but cannot know whether your assertions passed. An automatic fixture reports it. Put this in `tests/fixtures.ts` and import `test` and `expect` from it in the grid specs: ```typescript import { test as base, expect, type Page } from '@playwright/test'; function executor(action: string, args: Record): string { return `testingbot_executor: ${JSON.stringify({ action, arguments: args })}`; } export const test = base.extend<{ testingbotStatus: void }>({ testingbotStatus: [async ({ page }, use, testInfo) => { const remote = !!testInfo.project.use.connectOptions; if (remote) { await page.evaluate(() => {}, executor('setSessionName', { name: testInfo.title })); } await use(); if (remote) { await page.evaluate(() => {}, executor('setSessionStatus', { passed: testInfo.status === testInfo.expectedStatus, reason: testInfo.error?.message?.slice(0, 200) ?? '', })); } }, { auto: true }], }); export { expect }; ``` A visual test file for the grid, `tests/grid/todo.visual.spec.ts`: ```typescript import { test, expect } from '../fixtures'; test.beforeEach(async ({ page }) => { await page.goto('https://demo.playwright.dev/todomvc/'); }); test('empty todo app', async ({ page }) => { await expect(page).toHaveScreenshot('todo-empty.png'); }); test('todo list with two items', async ({ page }) => { const input = page.getByPlaceholder('What needs to be done?'); for (const item of ['Buy milk', 'Pay rent']) { await input.fill(item); await input.press('Enter'); } await expect(page.locator('.todoapp')).toHaveScreenshot('todo-two-items.png'); }); ``` Create the baselines, review them, then run again: ```bash export TB_KEY=your-key TB_SECRET=your-secret TB_BUILD=visual-grid npx playwright test -c playwright.testingbot.config.ts --update-snapshots=missing npx playwright test -c playwright.testingbot.config.ts ``` We ran this in Chrome and Firefox on Windows 11 and Safari on macOS Sequoia: six baselines in `tests/ __screenshots__ //`, and on the next runs all six tests passed and showed as passed sessions in the build in the TestingBot dashboard. The screenshot is rendered on the remote machine, so with the path template above it does not matter whether the runner is your laptop or a CI job: both compare against the same baseline for that browser and OS. How many sessions run at once depends on your plan; extra connections wait in a queue. More on the setup in our [Playwright Test integration guide](https://testingbot.com/support/web-automate/playwright/playwright-test). A team runs its `toHaveScreenshot()` suite against remote Windows browsers from both Mac laptops and Linux CI, using Playwright's default snapshot path. What goes wrong? Nothing: the remote browser is the same, so the files match Remote screenshots cannot be compared locally Laptops look for `-darwin` baselines and CI for `-linux` ones, so two copies of the same Windows baseline drift apart The comparison runs on the grid, so local baselines are ignored `{platform}` is the runner's operating system. With remote browsers it says nothing about how the page was rendered, so drop it and name baselines after the project. The comparison itself still happens locally in the Playwright runner. ## Option 2: TestingBot's visual comparison[#](https://testingbot.com#option-2-testingbot-s-visual-comparison) TestingBot can also take and compare the screenshot itself. You send a command with a snapshot name; the first time, TestingBot stores the screenshot as the baseline, and every later call with that name is compared with it on the server. The result comes back to your test within seconds, and each comparison appears in the member area with the baseline, the new screenshot and a diff image. Baselines live on TestingBot rather than in your repository, and approving a change is a click instead of a commit. From Playwright, the command is a `page.evaluate` call with a `visual.snapshot` action. Add a helper to `tests/fixtures.ts`: ```typescript type VisualResult = { match: boolean; pixelDifference?: number; url?: string; error?: string }; // TestingBot's server-side visual comparison (Chromium-based browsers over Playwright). // The project name goes into the snapshot name so each browser keeps its own baseline. export async function tbSnapshot(page: Page, name: string, options: Record = {}) { const fullName = `${name}-${test.info().project.name}`; return (await page.evaluate(() => {}, executor('visual.snapshot', { name: fullName, options }))) as VisualResult; } ``` and use it in a test: ```typescript import { test, expect, tbSnapshot } from '../fixtures'; // visual.snapshot over Playwright needs a Chromium-based browser (Chrome or Edge). test.skip(({ browserName }) => browserName !== 'chromium', 'Chromium-based browsers only'); test('todo app, compared by TestingBot', async ({ page }) => { await page.goto('https://demo.playwright.dev/todomvc/'); const input = page.getByPlaceholder('What needs to be done?'); for (const item of ['Buy milk', 'Pay rent']) { await input.fill(item); await input.press('Enter'); } const result = await tbSnapshot(page, 'todomvc-two-items', { ignoreSelectors: ['.info'], }); expect(result.error, result.error).toBeUndefined(); expect(result.match, `Visual difference, review it at ${result.url}`).toBe(true); }); ``` The first run returned `{ match: true, pixelDifference: 0 }` and created the baseline. Later runs return the comparison with links: ``` { match: false, pixelDifference: 874, visualId: ..., runId: ..., url: 'https://testingbot.com/members/visual//runs/' } ``` That failing result came from injecting the padding regression from chapter 1; the assertion message carries the link straight to the review page. Two things we learned running this: - **Over Playwright, it works with Chrome and Edge only.** In Firefox and Safari Playwright sessions the command returned `match: false` with an error saying it could not capture a screenshot, which is why the `expect(result.error)` line is there and why the test skips other engines. For Firefox and Safari, use the Selenium form below. - **Give every browser configuration its own snapshot name.** Put the browser and OS (here, the project name) into the name. When we reused one name across Firefox and Safari, Firefox was compared with Safari's screenshot and failed with about 19,600 different pixels on an unchanged page. With per-browser names, every repeat run matched. The options include `threshold` (per-pixel colour tolerance from 0 to 1, default 0.1), `antialiasing` (default on: anti-aliased pixels are not counted), `ignoreRegions` (rectangles in pixel coordinates), `ignoreSelectors` (CSS selectors to leave out) and `selector` (screenshot one element only). To replace a baseline from code, send `visual.baseline` with the same name. The [Playwright visual testing docs](https://testingbot.com/support/visual-testing/automated/playwright) list the options and the response fields. ### From Selenium, including Firefox and Safari[#](https://testingbot.com#from-selenium-including-firefox-and-safari) Selenium sends the same command through `executeScript`, and this works in Firefox and Safari too. This Node.js script ran green, repeatedly, in Firefox on Windows 11 and Safari on macOS Sequoia: ```javascript // node selenium-visual.mjs firefox WIN11 | node selenium-visual.mjs safari SEQUOIA import { Builder } from 'selenium-webdriver'; const [browserName = 'firefox', platformName = 'WIN11'] = process.argv.slice(2); const { TB_KEY, TB_SECRET } = process.env; const driver = await new Builder() .usingServer(`https://${TB_KEY}:${TB_SECRET}@hub.testingbot.com/wd/hub`) .withCapabilities({ browserName, browserVersion: 'latest', platformName, 'tb:options': { name: 'todo app visual check', build: process.env.TB_BUILD ?? 'visual-local' }, }) .build(); let passed = false; try { await driver.get('https://demo.playwright.dev/todomvc/'); // Put the browser and OS in the name: each configuration needs its own baseline. const name = `todomvc-home-${browserName}-${platformName}`.toLowerCase(); const command = JSON.stringify({ name, options: { threshold: 0.1 } }); const result = await driver.executeScript(`tb:visual.snapshot=${command}`); console.log(browserName, result); passed = result.match === true; } finally { await driver.executeScript(`tb:test-result=${passed ? 'passed' : 'failed'}`); await driver.quit(); } process.exitCode = passed ? 0 : 1; ``` The [Selenium visual testing docs](https://testingbot.com/support/visual-testing/automated/selenium) show the command in Java, Python, C#, Ruby and PHP, plus a `fullpage` option for Chrome, Edge and Firefox. There are equivalent pages for [Appium](https://testingbot.com/support/visual-testing/automated/appium) and [Puppeteer](https://testingbot.com/support/visual-testing/automated/puppeteer). A suite calls `tb:visual.snapshot=checkout` in Chrome, Firefox and Safari. Chrome passes, the others fail with large pixel differences on an unchanged page. What is the first thing to change? Raise `threshold` to 0.5 Include the browser and OS in the snapshot name, for example `checkout-firefox-win11` Call `visual.baseline` before every snapshot Run the browsers one after another instead of in parallel Each browser renders differently, so each needs its own baseline, the same principle as Playwright's per-project files. Unique names guarantee that. Re-baselining on every run would make the test pass by never comparing anything, and a high threshold hides real changes. ## Reviewing changes in the dashboard[#](https://testingbot.com#reviewing-changes-in-the-dashboard) Each comparison is a run under **Visual Testing** in the member area. A failed run shows the baseline, the new screenshot and a diff image highlighting the changed pixels, as described on [visual testing results](https://testingbot.com/support/visual-testing/results). From the run page you approve or reject it, and approving can make the new screenshot the baseline for future runs. A review queue lists the failed runs nobody has approved or rejected yet, so changes do not pile up unseen. This is the main trade-off against Option 1. With `toHaveScreenshot()`, baselines are files reviewed in pull requests, versioned with the code. With TestingBot's comparison, baselines are shared server-side state reviewed in a dashboard, which suits teams where designers or product owners approve changes without touching git. ## Option 3: scheduled visual tests without code[#](https://testingbot.com#option-3-scheduled-visual-tests-without-code) For pages you want to watch rather than test on every commit, such as marketing pages, documentation or a production checkout, TestingBot runs [codeless visual tests](https://testingbot.com/support/visual-testing/codeless): enter a URL, choose up to five browser configurations (each with a screen resolution), and a schedule. When a run differs from its baseline, you get an email or SMS alert with the baseline, screenshot and diff. Scheduled tests have the settings this course has been building towards, without code: - A **mismatch threshold** per test, 1% by default and adjustable between 0.1% and 5%, with a preview of how your last ten runs would have fared at the value you pick. - **Ignored regions** you draw over the baseline, and suggested regions for areas that changed in each of the last three runs. - A comparison that tolerates anti-aliasing: a pixel only counts when it differs beyond a tolerance and a neighbouring pixel changed too. - Automatic stabilisation before each screenshot: CSS animations frozen, lazy images loaded for full-page captures, the page's date and time frozen, the caret and scrollbars hidden, and on mobile devices the status bar removed. The [stabilisation guide](https://testingbot.com/support/visual-testing/stabilize-screenshots) has the details. A scheduled test can also run on demand from CI against a pull request's preview deployment, reporting back as a GitHub check, which the next chapter covers. ## Which option to choose[#](https://testingbot.com#which-option-to-choose) | | `toHaveScreenshot()` on the grid | TestingBot `visual.snapshot` | Scheduled visual tests | | --- | --- | --- | --- | | Baselines | PNG files in git | Stored on TestingBot | Stored on TestingBot | | Review | Pull request diff, HTML report | Member area run page | Member area, alerts | | Browsers | All Playwright engines | Chrome, Edge (Playwright); more with Selenium | Up to 5 per test | | Best for | Developer-owned UI tests | Shared review, existing Selenium suites | Monitoring live pages | Many teams combine them: component and page tests with `toHaveScreenshot()` in pull requests, and a scheduled test watching the production homepage and checkout. [Previous · Chapter 6 Cross-Browser Visual Testing](https://testingbot.com/resources/courses/visual-regression-testing/cross-browser-visual-testing) [Next · Chapter 8 Visual Regression Testing in CI with GitHub Actions](https://testingbot.com/resources/courses/visual-regression-testing/visual-tests-in-github-actions) On this page 1. [Option 1: your Playwright suite on remote browsers](https://testingbot.com#option-1-your-playwright-suite-on-remote-browsers) 2. [Reporting pass or fail to the dashboard](https://testingbot.com#reporting-pass-or-fail-to-the-dashboard) 3. [Option 2: TestingBot's visual comparison](https://testingbot.com#option-2-testingbot-s-visual-comparison) 4. [Reviewing changes in the dashboard](https://testingbot.com#reviewing-changes-in-the-dashboard) 5. [Option 3: scheduled visual tests without code](https://testingbot.com#option-3-scheduled-visual-tests-without-code) 6. [Which option to choose](https://testingbot.com#which-option-to-choose) Certification This course has a free, hands-on certificate. [How it works](https://testingbot.com/resources/courses/visual-regression-testing/certification)