Running API and UI Tests in CI and on a Cloud Grid
On this page
The suite now has two kinds of tests: API tests that never open a browser, and UI tests that do. In CI and on a cloud grid that difference decides what you install, what you pay for in sessions, and which machine has to be able to reach your application. This chapter starts with the question that explains all of it, "what runs where?", and then runs the browser half of the suite on TestingBot.
What runs where#
Playwright Test always runs your test code in Node.js worker processes on the machine where you typed npx playwright test: your laptop or a CI runner. When a project connects to a remote browser, only the browser moves. Requests are sent from different places depending on who sends them:
| Code | Where the HTTP request is sent from |
|---|---|
request fixture, playwright.request.newContext()
|
The test process: your machine or CI runner |
page.goto(), the page's own XHR and fetch calls |
The browser: the remote machine when you use a grid |
page.request, context.request
|
The browser's side as well: the remote machine |
You can see this for yourself with a test that asks a "what is my IP" service from all three:
import { browserTest as test } from '../fixtures';
const IP = 'https://api.ipify.org?format=json';
test('where does each request come from?', async ({ page, request }) => {
const fromTestProcess = await (await request.get(IP)).json();
const fromContext = await (await page.request.get(IP)).json();
await page.goto(IP);
const fromBrowser = JSON.parse(await page.locator('body').innerText());
console.log({ request: fromTestProcess.ip, 'page.request': fromContext.ip, 'page.goto': fromBrowser.ip });
});
Run locally, all three print the same address. Run against a Chrome on Windows on TestingBot, request still printed the local machine's address, while page.request and page.goto both printed the address of the remote machine.
Three consequences:
-
API tests need no browser and open no session. The
apiproject runs in your CI job's Node process whether or not the UI projects use a grid. It costs no grid minutes and needs nonpx playwright install. -
Both sides must reach the application. Chapter 4's fixtures create data from your machine, and the remote browser then reads it. For a public site that is automatic. For a staging environment behind a firewall, your CI runner can reach the API directly, but the remote browser needs a route in: that is what TestingBot Tunnel is for.
page.requestcalls would go through that same remote side. -
page.requestis not a shortcut around the tunnel. If you use it for setup, it is subject to the same network as the browser. Theapifixture, which usesplaywright.request.newContext(), is not.
Your staging API is only reachable from the office network. CI runs in the office; the browsers run on a cloud grid. The api fixture creates data fine, but the UI tests cannot load the staging site. What is going on?
Test code, including playwright.request contexts, runs where you started the run. Page loads happen on the remote machine, which needs TestingBot Tunnel to reach a private network. Switching to page.request would move the API calls outside as well.
Report results to TestingBot, but only from browser tests#
A TestingBot session shows a video and logs, but it cannot see your assertions. As in the Playwright course's grid chapter, an automatic fixture reports the test's name and result with a testingbot_executor call through page.evaluate. In this suite there is a twist: the fixture needs page, and an automatic fixture that needs page makes every test that uses it start a browser, including pure API tests. On the grid config below, the api project has no remote endpoint, so those tests would launch a local browser, and in a CI job that skipped npx playwright install they would fail outright.
So the status fixture goes into its own layer. In tests/fixtures.ts, add executor and browserTest below test, and replace the uiTest declaration from chapter 3 with the version below, which builds on browserTest instead of test (keeping both would declare uiTest twice and fail to compile). The end of the file then reads:
function executor(action: string, args: Record<string, unknown>): string {
return `testingbot_executor: ${JSON.stringify({ action, arguments: args })}`;
}
// For tests that use a browser: report name and result to TestingBot.
// Kept out of `test` on purpose: an auto fixture that needs `page` would
// start a browser (and a remote session) for every API-only test too.
export const browserTest = test.extend<{ testingbotStatus: void }>({
testingbotStatus: [async ({ page }, use, testInfo) => {
const remote = !!testInfo.project.use.connectOptions;
if (remote) {
await page.evaluate(() => {}, executor('setSessionName', { name: testInfo.title }));
}
await use();
if (remote) {
const passed = testInfo.status === testInfo.expectedStatus;
await page.evaluate(() => {}, executor('setSessionStatus', {
passed,
reason: testInfo.error?.message?.slice(0, 200) ?? '',
}));
}
}, { auto: true }],
});
// Replaces the chapter 3 version: same storageState override, now on top of browserTest.
export const uiTest = browserTest.extend({
storageState: async ({ token }, use) => {
await use({
cookies: [],
origins: [{ origin: UI_URL, localStorage: [{ name: 'auth-token', value: token }] }],
});
},
});
export { expect };
The fixtures file now has three layers: test for API tests, browserTest for browser tests that need no login, and uiTest for browser tests of the logged-in shop. Change the first line of tests/ui/mocking.spec.ts to import from it:
import { browserTest as test, expect } from '../fixtures';
favorites.spec.ts and login.spec.ts already use uiTest, so they pick up the status fixture without changes. The setSessionName and setSessionStatus actions are documented in the Playwright Test integration guide.
The grid config#
Add the endpoint helper from the Playwright course as testingbot.ts, with the build name read from an environment variable:
type TestingBotBrowser = {
browserName: 'chrome' | 'edge' | 'firefox' | 'safari';
browserVersion?: string;
platform?: string;
};
export function testingbotEndpoint(browser: TestingBotBrowser): string {
const capabilities = {
...browser,
'tb:options': {
key: process.env.TB_KEY,
secret: process.env.TB_SECRET,
build: process.env.TB_BUILD ?? 'playwright-api-testing',
},
};
return `wss://cloud.testingbot.com/playwright?capabilities=${encodeURIComponent(JSON.stringify(capabilities))}`;
}
Then playwright.testingbot.config.ts, which keeps the api project exactly as it is and replaces the two local browser projects with remote ones:
import { defineConfig } from '@playwright/test';
import baseConfig from './playwright.config';
import { testingbotEndpoint } from './testingbot';
import { UI_URL } from './lib/env';
const apiProject = baseConfig.projects!.find((p) => p.name === 'api')!;
export default defineConfig({
...baseConfig,
workers: 2,
projects: [
apiProject, // no browser: runs in the local Node process, opens no session
{
name: 'chrome-windows',
testDir: './tests/ui',
use: {
browserName: 'chromium',
baseURL: UI_URL,
connectOptions: {
wsEndpoint: testingbotEndpoint({ browserName: 'chrome', browserVersion: 'latest', platform: 'WIN11' }),
},
},
},
{
name: 'firefox-linux',
testDir: './tests/ui',
use: {
browserName: 'firefox',
baseURL: UI_URL,
connectOptions: {
wsEndpoint: testingbotEndpoint({ browserName: 'firefox', browserVersion: 'latest', platform: 'LINUX' }),
},
},
},
],
});
browserName still has to name the Playwright engine that matches the remote browser (chromium for Chrome and Edge, firefox for Firefox, webkit for Safari). The Playwright capabilities generator lists the browser and platform values you can use.
Run it with your key and secret from the TestingBot member area:
export TB_KEY=your-key TB_SECRET=your-secret
npx playwright test -c playwright.testingbot.config.ts --workers=1
✓ 12 [api] › tests/api/schema.spec.ts:25:1 › zod explains what is wrong (1ms)
✓ 13 [chrome-windows] › tests/ui/favorites.spec.ts:23:1 › a favorite created through the API is listed (3.0s)
✓ 14 [chrome-windows] › tests/ui/favorites.spec.ts:30:1 › removing a favorite in the UI removes it on the server (2.9s)
✓ 15 [chrome-windows] › tests/ui/login.spec.ts:3:1 › starts logged in with an API token (2.0s)
...
✓ 24 [firefox-linux] › tests/ui/mocking.spec.ts:36:1 › a failing search looks like an empty one (2.0s)
24 passed (1.1m)
All twelve API tests ran locally first, in well under a second each. The twelve UI tests ran on Chrome 156 on Windows 11 and Firefox 158 on Linux, with their data created by the local api fixture and their routes and mocks applied inside the remote browsers.
Sessions follow workers, not tests#
On the TestingBot dashboard that run shows two sessions, not twelve. Playwright connects one remote browser per worker and reuses it for every test that worker runs in the project, with a fresh context per test. With --workers=1, each browser project ran in one worker and therefore one session. The session's name is the last test's title, because each test renames it; its status is the last reported result. A failing test also restarts the worker, so a failure shows up as an extra session.
More workers means more parallel sessions. How many can run at once depends on your plan; connections over the limit wait in a queue until a slot frees up. The API project never counts against it.
A suite has 40 API tests and 10 UI tests on one remote browser project, run with --workers=2. Roughly how many TestingBot sessions does a green run create?
API tests never open a browser, so they create no sessions. Remote browsers are opened per worker and reused across that worker's tests, so the UI tests share at most two sessions.
A note on public demo sites: in one grid run, a UI test that logged in through the Toolshop's form landed on a "Performing security verification" page from the site's bot protection instead of the account page, while the tests that started with an API token were not affected. If a trace from a remote run shows such a page, the problem is the site's protection, not your test. For your own application, ask your team to allow the test traffic.
CI: two jobs#
The split between API and UI tests maps naturally onto two CI jobs. A GitHub Actions workflow, building on the one from chapter 7 of the Playwright course:
name: Tests
on:
push:
branches: [ main ]
pull_request:
jobs:
api:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: actions/setup-node@v4
with:
node-version: lts/*
- run: npm ci
# No "npx playwright install": API tests never start a browser.
- run: npx playwright test --project=api
ui:
needs: api
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: actions/setup-node@v4
with:
node-version: lts/*
- run: npm ci
- name: UI tests on TestingBot
run: >
npx playwright test -c playwright.testingbot.config.ts
--project=chrome-windows --project=firefox-linux
env:
TB_KEY: ${{ secrets.TB_KEY }}
TB_SECRET: ${{ secrets.TB_SECRET }}
TB_BUILD: ci-${{ github.run_id }}
- uses: actions/upload-artifact@v4
if: ${{ !cancelled() }}
with:
name: playwright-report
path: playwright-report/
retention-days: 14
-
The
apijob installs no browsers. That step is usually the slowest part of a Playwright job, and these tests do not need it. RunningPLAYWRIGHT_BROWSERS_PATH=/nonexistent npx playwright test --project=api, as in chapter 1, is a quick local check that nothing in the API project accidentally asks for a browser. -
needs: apimakes the UI job wait. If the API is broken, UI tests that build their data through it would fail for the same reason, more slowly and with less useful errors. Failing fast on the API job saves grid time and points at the cause. -
The UI job installs no browsers either, because both of its projects connect to TestingBot. If you also keep local browser projects in CI, add
npx playwright install --with-depsback for them. -
TB_BUILDgroups every session of one workflow run into one build on the dashboard. StoreTB_KEYandTB_SECRETas repository secrets.
After a refactor, the CI job that runs only --project=api without installing browsers fails with "Executable doesn't exist". What most likely changed?
The request fixture never needs a browser. A browser launch in an API-only run means some fixture in the chain asks for page, context or browser, and an auto fixture does that for every test that imports it.
For more on the TestingBot side, including every capability and other language bindings, see the Playwright documentation and the Playwright options reference.