Run AI-Written Suites on TestingBot

Generated or hand-written, a reviewed Playwright suite is just a Playwright suite. Running it on a cloud grid needs no AI at all, and that is how it should be: the run that decides whether you ship should be deterministic code against real browsers. This chapter puts the four reviewed TodoMVC tests on TestingBot across two browser and OS combinations, reports pass or fail to the dashboard, and then looks at what the TestingBot MCP server adds for an agent, and what it does not do.

Point the suite at the grid#

Playwright connects to remote browsers over a WebSocket. The cloud grid chapter of the Playwright course explains the mechanism in depth; here is the short version used for this course. A helper builds TestingBot's endpoint URL from the browser you want:

type TestingBotBrowser = {
  browserName: 'chrome' | 'edge' | 'firefox' | 'safari';
  browserVersion?: string;
  platform?: string;
};

export function testingbotEndpoint(browser: TestingBotBrowser): string {
  const capabilities = {
    ...browser,
    'tb:options': {
      key: process.env.TB_KEY,
      secret: process.env.TB_SECRET,
      build: process.env.TB_BUILD ?? 'ai-test-automation-local',
    },
  };
  return `wss://cloud.testingbot.com/playwright?capabilities=${encodeURIComponent(JSON.stringify(capabilities))}`;
}

A second config, playwright.testingbot.config.ts, reuses the local one and replaces the projects:

import { defineConfig } from '@playwright/test';
import baseConfig from './playwright.config';
import { testingbotEndpoint } from './testingbot';

export default defineConfig({
  ...baseConfig,
  testIgnore: /seed\.spec\.ts/,
  workers: 2,
  projects: [
    {
      name: 'chrome-windows',
      use: {
        browserName: 'chromium',
        connectOptions: {
          wsEndpoint: testingbotEndpoint({ browserName: 'chrome', browserVersion: 'latest', platform: 'WIN11' }),
        },
      },
    },
    {
      name: 'firefox-linux',
      use: {
        browserName: 'firefox',
        connectOptions: {
          wsEndpoint: testingbotEndpoint({ browserName: 'firefox', browserVersion: 'latest', platform: 'LINUX' }),
        },
      },
    },
  ],
});

testIgnore keeps the agents' seed test out of grid runs: it is setup for the agents, not a test of the app. The project's browserName must match the remote engine (chrome and edge use chromium, safari uses webkit). The Playwright capabilities generator lists valid browser and platform combinations.

Report pass or fail#

TestingBot cannot see your assertions. The test tells it the result through a testingbot_executor call, which is easiest as an automatic fixture in tests/fixtures.ts:

import { test as base, expect } from '@playwright/test';

function executor(action: string, args: Record<string, unknown>): string {
  return `testingbot_executor: ${JSON.stringify({ action, arguments: args })}`;
}

export const test = base.extend<{ testingbotStatus: void }>({
  testingbotStatus: [async ({ page }, use, testInfo) => {
    const remote = !!testInfo.project.use.connectOptions;
    if (remote) {
      await page.evaluate(() => {}, executor('setSessionName', { name: testInfo.title }));
    }
    await use();
    if (remote) {
      const passed = testInfo.status === testInfo.expectedStatus;
      await page.evaluate(() => {}, executor('setSessionStatus', {
        passed,
        reason: testInfo.error?.message?.slice(0, 200) ?? '',
      }));
    }
  }, { auto: true }],
});

export { expect };

The generated test files then import test and expect from '../fixtures' instead of '@playwright/test'. That one-line change in each file is the kind of edit an agent does well. Ask for it, and check the diff touches nothing else. The actions are documented in the Playwright Test integration guide.

Run it#

export TB_KEY=your-key TB_SECRET=your-secret TB_BUILD=ai-test-automation-course-verify
npx playwright test -c playwright.testingbot.config.ts --reporter=list

Our run on 12 November 2026:

Running 8 tests using 2 workers

  ✓  2 [chrome-windows] › tests/todomvc/clear-completed.spec.ts:7:3 › TodoMVC Core Functionality › Clear completed todos (3.8s)
  ✓  1 [chrome-windows] › tests/todomvc/add-todos.spec.ts:7:3 › TodoMVC Core Functionality › Add multiple todos (3.0s)
  ✓  3 [chrome-windows] › tests/todomvc/complete-todo.spec.ts:7:3 › TodoMVC Core Functionality › Complete a single todo (3.3s)
  ✓  4 [chrome-windows] › tests/todomvc/filter-active.spec.ts:7:3 › TodoMVC Core Functionality › Active filter shows only incomplete todos (3.0s)
  ✓  6 [firefox-linux] › tests/todomvc/clear-completed.spec.ts:7:3 › TodoMVC Core Functionality › Clear completed todos (4.2s)
  ✓  7 [firefox-linux] › tests/todomvc/complete-todo.spec.ts:7:3 › TodoMVC Core Functionality › Complete a single todo (2.6s)
  ✓  8 [firefox-linux] › tests/todomvc/filter-active.spec.ts:7:3 › TodoMVC Core Functionality › Active filter shows only incomplete todos (3.3s)
  ✓  5 [firefox-linux] › tests/todomvc/add-todos.spec.ts:7:3 › TodoMVC Core Functionality › Add multiple todos (4.6s)

  8 passed (3.5m)

The dashboard showed four sessions, not eight: Chrome 156 on Windows 11 and Firefox 158 on Linux, two each, all marked passed. That is Playwright's worker model, not a grid quirk. Each worker connects once per project and runs several tests in that browser, each in a fresh context. The session name and status are set by whichever test ran last in that session.

Does that mean a failure can be hidden by a later pass in the same session? No. When a test fails, Playwright discards that worker and starts a new one for the remaining tests, so the failed test is always the last one in its session and its status sticks. The number of sessions follows the number of workers, not the number of tests: one worker running both projects would give you two sessions for all eight tests.

The wall time (3.5 minutes for 8 tests that each take about 3 seconds) is mostly session start-up. More workers, up to your plan's parallel limit, shorten it.

What the TestingBot MCP server does#

@testingbot/mcp-server (1.2.0 at the time of writing) gives an MCP client tools for your TestingBot account. Based on its README and tool definitions, they fall into four groups:

• Account and results: getTests, getTestDetails, updateTest (name, status, build), getBuilds, getTestsForBuild, getFailureLogs, plus storage, team and user tools.
• Driving a remote browser or device: it bundles @testingbot/automation-mcp, whose tb_openBrowser, tb_navigate, tb_snapshot, tb_click, tb_type, tb_getText, tb_screenshot and tb_closeBrowser tools drive a TestingBot browser over WebDriver, and return a live-view URL so you can watch. Real-device sessions go through a bundled Appium MCP server.
• Running native mobile suites: Maestro flows, Espresso and XCUITest runs, with tools to upload apps, start runs and fetch results.
• Other: cross-browser screenshots, starting a live (manual) session, CDP sessions, and starting or stopping a TestingBot tunnel so remote browsers can reach localhost.

Notice what is not there: a tool that runs your Playwright suite. The suite runs from your machine or CI with npx playwright test, as above. The MCP server is for what an agent does around that run: exploring a page on a browser you do not have locally (Safari on macOS from a Linux laptop), looking up results, reading logs, renaming or re-marking sessions, starting a tunnel.

The tb_* browser tools are worth knowing as the remote counterpart of chapter 1. When an agent explores with Playwright MCP on your laptop, it sees your local Chromium. Pointing the same exploration at a real Safari on TestingBot can reveal what a WebKit-only bug looks like before you write the test for it. The README's caveat applies: sessions are billed test minutes, idle browser sessions are closed after four minutes by default, and the live-view URL contains an access hash, so do not paste it anywhere public.

We used the server in Claude Code to look up this chapter's run. getFailureLogs worked as described in chapter 5. Finding the build by name did not: in version 1.2.0, getBuilds returned build IDs and creation times but not the build names we had set, so the agent could not tell which build was ours. Looking up by session ID (from the dashboard or getTests) works. Agents tend to keep calling tools when an answer is not there, so if a lookup fails, tell yours to stop and say so rather than page through two hundred builds.

Before the certificate#

The practical task for this course's certificate is this chapter: at least three agent-generated, human-reviewed tests, passing on the TestingBot grid on at least two configurations, in the build certificate-ai-test-automation, with status reported through the fixture above. Set TB_BUILD=certificate-ai-test-automation and run the grid config.