--- title: Run AI-Written Suites on TestingBot | TestingBot description: Generated or hand-written, a reviewed Playwright suite is just a Playwright suite. Running it on a cloud grid needs no AI at all, and that is how it should… source_url: html: https://testingbot.com/resources/courses/ai-test-automation/run-on-testingbot md: https://testingbot.com/resources/courses/ai-test-automation/run-on-testingbot.md --- Course [AI Test Automation: A Hands-On Course](https://testingbot.com/resources/courses/ai-test-automation) Chapter 7 of 9 1. [1 AI Test Automation: A Hands-On Course](https://testingbot.com/resources/courses/ai-test-automation) 2. [2 Set Up an Agent with Playwright MCP](https://testingbot.com/resources/courses/ai-test-automation/set-up-playwright-mcp) 3. [3 From Test Plan to Playwright Test with an Agent](https://testingbot.com/resources/courses/ai-test-automation/generate-tests-from-a-plan) 4. [4 Playwright Test Agents: Planner, Generator and Healer](https://testingbot.com/resources/courses/ai-test-automation/playwright-test-agents) 5. [5 Self-Healing Tests: When Healing Hides a Regression](https://testingbot.com/resources/courses/ai-test-automation/self-healing-tests) 6. [6 AI for Failure Triage: Traces, Logs and Privacy](https://testingbot.com/resources/courses/ai-test-automation/ai-failure-triage) 7. [7 Run AI-Written Suites on TestingBot](https://testingbot.com/resources/courses/ai-test-automation/run-on-testingbot) 8. [8 Guardrails for AI in a Test Suite](https://testingbot.com/resources/courses/ai-test-automation/guardrails) 9. [9 AI Test Automation Quiz](https://testingbot.com/resources/courses/ai-test-automation/knowledge-check) [Exit course](https://testingbot.com/resources/courses) Chapter 7 of 9 Intermediate 8 min read 2 quick checks Updated Nov 2026 # Run AI-Written Suites on TestingBot On this page 1. [Point the suite at the grid](https://testingbot.com#point-the-suite-at-the-grid) 2. [Report pass or fail](https://testingbot.com#report-pass-or-fail) 3. [Run it](https://testingbot.com#run-it) 4. [What the TestingBot MCP server does](https://testingbot.com#what-the-testingbot-mcp-server-does) 5. [Before the certificate](https://testingbot.com#before-the-certificate) Generated or hand-written, a reviewed Playwright suite is just a Playwright suite. Running it on a cloud grid needs no AI at all, and that is how it should be: the run that decides whether you ship should be deterministic code against real browsers. This chapter puts the four reviewed TodoMVC tests on TestingBot across two browser and OS combinations, reports pass or fail to the dashboard, and then looks at what the TestingBot MCP server adds for an agent, and what it does not do. ## Point the suite at the grid[#](https://testingbot.com#point-the-suite-at-the-grid) Playwright connects to remote browsers over a WebSocket. The [cloud grid chapter of the Playwright course](https://testingbot.com/resources/courses/playwright/run-on-a-cloud-grid) explains the mechanism in depth; here is the short version used for this course. A helper builds TestingBot's endpoint URL from the browser you want: ```typescript type TestingBotBrowser = { browserName: 'chrome' | 'edge' | 'firefox' | 'safari'; browserVersion?: string; platform?: string; }; export function testingbotEndpoint(browser: TestingBotBrowser): string { const capabilities = { ...browser, 'tb:options': { key: process.env.TB_KEY, secret: process.env.TB_SECRET, build: process.env.TB_BUILD ?? 'ai-test-automation-local', }, }; return `wss://cloud.testingbot.com/playwright?capabilities=${encodeURIComponent(JSON.stringify(capabilities))}`; } ``` A second config, `playwright.testingbot.config.ts`, reuses the local one and replaces the projects: ```typescript import { defineConfig } from '@playwright/test'; import baseConfig from './playwright.config'; import { testingbotEndpoint } from './testingbot'; export default defineConfig({ ...baseConfig, testIgnore: /seed\.spec\.ts/, workers: 2, projects: [ { name: 'chrome-windows', use: { browserName: 'chromium', connectOptions: { wsEndpoint: testingbotEndpoint({ browserName: 'chrome', browserVersion: 'latest', platform: 'WIN11' }), }, }, }, { name: 'firefox-linux', use: { browserName: 'firefox', connectOptions: { wsEndpoint: testingbotEndpoint({ browserName: 'firefox', browserVersion: 'latest', platform: 'LINUX' }), }, }, }, ], }); ``` `testIgnore` keeps the agents' seed test out of grid runs: it is setup for the agents, not a test of the app. The project's `browserName` must match the remote engine (`chrome` and `edge` use `chromium`, `safari` uses `webkit`). The [Playwright capabilities generator](https://testingbot.com/support/web-automate/playwright/capabilities) lists valid browser and platform combinations. ## Report pass or fail[#](https://testingbot.com#report-pass-or-fail) TestingBot cannot see your assertions. The test tells it the result through a `testingbot_executor` call, which is easiest as an automatic fixture in `tests/fixtures.ts`: ```typescript import { test as base, expect } from '@playwright/test'; function executor(action: string, args: Record): string { return `testingbot_executor: ${JSON.stringify({ action, arguments: args })}`; } export const test = base.extend<{ testingbotStatus: void }>({ testingbotStatus: [async ({ page }, use, testInfo) => { const remote = !!testInfo.project.use.connectOptions; if (remote) { await page.evaluate(() => {}, executor('setSessionName', { name: testInfo.title })); } await use(); if (remote) { const passed = testInfo.status === testInfo.expectedStatus; await page.evaluate(() => {}, executor('setSessionStatus', { passed, reason: testInfo.error?.message?.slice(0, 200) ?? '', })); } }, { auto: true }], }); export { expect }; ``` The generated test files then import `test` and `expect` from `'../fixtures'` instead of `'@playwright/test'`. That one-line change in each file is the kind of edit an agent does well. Ask for it, and check the diff touches nothing else. The actions are documented in the [Playwright Test integration guide](https://testingbot.com/support/web-automate/playwright/playwright-test). Your grid run is green in the terminal, but the TestingBot dashboard shows every session without a pass or fail status. What is missing? The `build` option in `tb:options` `trace: 'on'` in the config The tests are not using the fixture that sends `setSessionStatus`, for example because they still import from `@playwright/test` A TestingBot tunnel The status only reaches TestingBot through the executor call. A generated file that still imports `test` from `@playwright/test` never runs the automatic fixture. The build name groups sessions but does not set their result. ## Run it[#](https://testingbot.com#run-it) ```bash export TB_KEY=your-key TB_SECRET=your-secret TB_BUILD=ai-test-automation-course-verify npx playwright test -c playwright.testingbot.config.ts --reporter=list ``` Our run on 12 November 2026: ``` Running 8 tests using 2 workers ✓ 2 [chrome-windows] › tests/todomvc/clear-completed.spec.ts:7:3 › TodoMVC Core Functionality › Clear completed todos (3.8s) ✓ 1 [chrome-windows] › tests/todomvc/add-todos.spec.ts:7:3 › TodoMVC Core Functionality › Add multiple todos (3.0s) ✓ 3 [chrome-windows] › tests/todomvc/complete-todo.spec.ts:7:3 › TodoMVC Core Functionality › Complete a single todo (3.3s) ✓ 4 [chrome-windows] › tests/todomvc/filter-active.spec.ts:7:3 › TodoMVC Core Functionality › Active filter shows only incomplete todos (3.0s) ✓ 6 [firefox-linux] › tests/todomvc/clear-completed.spec.ts:7:3 › TodoMVC Core Functionality › Clear completed todos (4.2s) ✓ 7 [firefox-linux] › tests/todomvc/complete-todo.spec.ts:7:3 › TodoMVC Core Functionality › Complete a single todo (2.6s) ✓ 8 [firefox-linux] › tests/todomvc/filter-active.spec.ts:7:3 › TodoMVC Core Functionality › Active filter shows only incomplete todos (3.3s) ✓ 5 [firefox-linux] › tests/todomvc/add-todos.spec.ts:7:3 › TodoMVC Core Functionality › Add multiple todos (4.6s) 8 passed (3.5m) ``` The dashboard showed **four** sessions, not eight: Chrome 156 on Windows 11 and Firefox 158 on Linux, two each, all marked passed. That is Playwright's worker model, not a grid quirk. Each worker connects once per project and runs several tests in that browser, each in a fresh context. The session name and status are set by whichever test ran last in that session. Does that mean a failure can be hidden by a later pass in the same session? No. When a test fails, Playwright discards that worker and starts a new one for the remaining tests, so the failed test is always the last one in its session and its status sticks. The number of sessions follows the number of workers, not the number of tests: one worker running both projects would give you two sessions for all eight tests. The wall time (3.5 minutes for 8 tests that each take about 3 seconds) is mostly session start-up. More workers, up to your plan's parallel limit, shorten it. ## What the TestingBot MCP server does[#](https://testingbot.com#what-the-testingbot-mcp-server-does) `@testingbot/mcp-server` (1.2.0 at the time of writing) gives an MCP client tools for your TestingBot account. Based on its README and tool definitions, they fall into four groups: - **Account and results** : `getTests`, `getTestDetails`, `updateTest` (name, status, build), `getBuilds`, `getTestsForBuild`, `getFailureLogs`, plus storage, team and user tools. - **Driving a remote browser or device** : it bundles `@testingbot/automation-mcp`, whose `tb_openBrowser`, `tb_navigate`, `tb_snapshot`, `tb_click`, `tb_type`, `tb_getText`, `tb_screenshot` and `tb_closeBrowser` tools drive a TestingBot browser over WebDriver, and return a live-view URL so you can watch. Real-device sessions go through a bundled Appium MCP server. - **Running native mobile suites** : Maestro flows, Espresso and XCUITest runs, with tools to upload apps, start runs and fetch results. - **Other** : cross-browser screenshots, starting a live (manual) session, CDP sessions, and starting or stopping a TestingBot tunnel so remote browsers can reach `localhost`. Notice what is not there: a tool that runs your Playwright suite. The suite runs from your machine or CI with `npx playwright test`, as above. The MCP server is for what an agent does around that run: exploring a page on a browser you do not have locally (Safari on macOS from a Linux laptop), looking up results, reading logs, renaming or re-marking sessions, starting a tunnel. The `tb_*` browser tools are worth knowing as the remote counterpart of chapter 1. When an agent explores with Playwright MCP on your laptop, it sees your local Chromium. Pointing the same exploration at a real Safari on TestingBot can reveal what a WebKit-only bug looks like before you write the test for it. The README's caveat applies: sessions are billed test minutes, idle browser sessions are closed after four minutes by default, and the live-view URL contains an access hash, so do not paste it anywhere public. We used the server in Claude Code to look up this chapter's run. `getFailureLogs` worked as described in chapter 5. Finding the build _by name_ did not: in version 1.2.0, `getBuilds` returned build IDs and creation times but not the build names we had set, so the agent could not tell which build was ours. Looking up by session ID (from the dashboard or `getTests`) works. Agents tend to keep calling tools when an answer is not there, so if a lookup fails, tell yours to stop and say so rather than page through two hundred builds. You ask your agent, connected to the TestingBot MCP server, to "run our Playwright suite on Safari". What should you expect? The server uploads your tests and runs them with `npx playwright test` on TestingBot There is no tool for that; the agent can run `npx playwright test` with a grid config itself, or drive a Safari session interactively through the `tb_*` tools The request fails because the server only supports mobile apps The MCP server manages the account and drives interactive sessions; your Playwright suite still runs through Playwright's own connect mechanism. Read the tool list of any MCP server before assuming what it can do, because agents will try to improvise when a tool is missing. ## Before the certificate[#](https://testingbot.com#before-the-certificate) The practical task for this course's certificate is this chapter: at least three agent-generated, human-reviewed tests, passing on the TestingBot grid on at least two configurations, in the build `certificate-ai-test-automation`, with status reported through the fixture above. Set `TB_BUILD=certificate-ai-test-automation` and run the grid config. [Previous · Chapter 6 AI for Failure Triage: Traces, Logs and Privacy](https://testingbot.com/resources/courses/ai-test-automation/ai-failure-triage) [Next · Chapter 8 Guardrails for AI in a Test Suite](https://testingbot.com/resources/courses/ai-test-automation/guardrails) On this page 1. [Point the suite at the grid](https://testingbot.com#point-the-suite-at-the-grid) 2. [Report pass or fail](https://testingbot.com#report-pass-or-fail) 3. [Run it](https://testingbot.com#run-it) 4. [What the TestingBot MCP server does](https://testingbot.com#what-the-testingbot-mcp-server-does) 5. [Before the certificate](https://testingbot.com#before-the-certificate) Certification This course has a free, hands-on certificate. [How it works](https://testingbot.com/resources/courses/ai-test-automation/certification)