Accessibility Testing on TestingBot
On this page
So far the scans ran in one engine on one machine. Your users do not: they have Safari on a Mac, Firefox on Windows, Chrome on whatever they own, and accessibility results are not always identical across engines. This chapter runs the course suite on TestingBot's browser grid, then looks at two TestingBot features built for accessibility: the tb:accessibility command for Selenium tests, and scheduled accessibility scans that need no test code at all.
For the product overview, see automated accessibility testing; this chapter is the hands-on part.
The suite on Chrome, Firefox and Safari#
Playwright connects to a remote browser over a WebSocket, and @axe-core/playwright works the same way on that remote page, because it injects axe through the normal Playwright protocol. Create testingbot.ts:
type TestingBotBrowser = {
browserName: 'chrome' | 'edge' | 'firefox' | 'safari';
browserVersion?: string;
platform?: string;
};
export function testingbotEndpoint(browser: TestingBotBrowser): string {
const capabilities = {
browserVersion: 'latest',
...browser,
'tb:options': {
key: process.env.TB_KEY,
secret: process.env.TB_SECRET,
build: process.env.TB_BUILD ?? 'accessibility-local',
},
};
return `wss://cloud.testingbot.com/playwright?capabilities=${encodeURIComponent(JSON.stringify(capabilities))}`;
}
and playwright.testingbot.config.ts, which reuses the local config and swaps the projects:
import { defineConfig } from '@playwright/test';
import baseConfig from './playwright.config';
import { testingbotEndpoint } from './testingbot';
export default defineConfig({
...baseConfig,
workers: 1,
projects: [
{
name: 'chrome-windows',
use: {
browserName: 'chromium',
connectOptions: { wsEndpoint: testingbotEndpoint({ browserName: 'chrome', platform: 'WIN11' }) },
},
},
{
name: 'firefox-windows',
use: {
browserName: 'firefox',
connectOptions: { wsEndpoint: testingbotEndpoint({ browserName: 'firefox', platform: 'WIN11' }) },
},
},
{
name: 'safari-macos',
use: {
browserName: 'webkit',
connectOptions: { wsEndpoint: testingbotEndpoint({ browserName: 'safari', platform: 'SEQUOIA' }) },
},
},
],
});
browserName in each project must match the remote browser's engine: chromium for Chrome and Edge, firefox for Firefox, webkit for Safari. The Playwright capabilities generator lists the platform names, and the options reference documents build and every other option. workers: 1 keeps one session at a time, which suits a trial account; raise it to what your plan allows.
Reporting pass or fail#
TestingBot records a video and logs for every session, but it cannot see your assertions. Add an automatic fixture to tests/axe.ts that names each session and reports its result:
type AxeFixtures = { makeAxeBuilder: () => AxeBuilder; testingbotStatus: void };
function executor(action: string, args: Record<string, unknown>): string {
return `testingbot_executor: ${JSON.stringify({ action, arguments: args })}`;
}
export const test = base.extend<AxeFixtures>({
makeAxeBuilder: async ({ page }, use) => {
// ... unchanged from chapter 4
},
// On TestingBot, name each session and mark it passed or failed. A no-op locally.
testingbotStatus: [async ({ page }, use, testInfo) => {
const remote = !!testInfo.project.use.connectOptions;
if (remote) await page.evaluate(() => {}, executor('setSessionName', { name: testInfo.title }));
await use();
if (remote) {
await page.evaluate(() => {}, executor('setSessionStatus', {
passed: testInfo.status === testInfo.expectedStatus,
reason: testInfo.error?.message?.slice(0, 200) ?? '',
}));
}
}, { auto: true }],
});
The setSessionName and setSessionStatus actions are documented in the Playwright Test guide. Every spec that imports test from ./axe now reports its status. The reflow spec from chapter 1 imported from @playwright/test; switch its import to ./axe too.
Run it:
export TB_KEY=your-key TB_SECRET=your-secret TB_BUILD=course-a11y-playwright
npx playwright test -c playwright.testingbot.config.ts
What the other engines found#
The first run on the grid had two failures, both in Safari, and both are real lessons rather than flakes:
-
The broken sign-up form reported a sixth rule in Safari. WebKit rendered the text-less submit button at 32 by 23.3 pixels, so
target-size(WCAG 2.5.8) fired. Chromium and Firefox render it at 24 pixels or taller. Chapter 5's test usesarrayContainingbecause of this. -
The keyboard test from chapter 8 could not tab to the checkbox in Safari. Safari only moves
Tabfocus between text fields by default; reaching buttons, checkboxes and links needsOption+Tab. The test now pressesAlt+Tab(Playwright's name for Option) whenbrowserNameiswebkit.
After those changes the whole suite passes in all three browsers: 27 tests across Chrome and Firefox on Windows 11 and Safari on macOS Sequoia, one session at a time, in about three minutes. A suite that only ever ran in Chromium would never have shown either difference.
An axe scan passes in Chromium but reports target-size in Safari on the same page. What should you conclude?
axe measures the rendered layout, and engines lay out default controls differently. The violation describes what Safari users actually get. target-size is a WCAG 2.2 AA rule, not a best practice.
tb:accessibility in Selenium tests#
For Selenium suites, TestingBot provides a custom command that runs an accessibility check on the current page and returns the results to your test, with no axe package in your project. It works in every Selenium language, because it is just an executeScript call. The Selenium accessibility guide has examples for Java, Node.js, C#, Python, Ruby and PHP.
Added to the JUnit class from chapter 3 (it also needs import org.junit.jupiter.api.Assumptions;):
@Test
@SuppressWarnings("unchecked")
void testingbotScanFindsNoWcagViolations() {
Assumptions.assumeTrue(onTestingBot, "tb:accessibility only exists on the TestingBot grid");
driver.get("https://the-internet.herokuapp.com/login");
Map<String, Object> report =
(Map<String, Object>) ((JavascriptExecutor) driver).executeScript("tb:accessibility");
List<Map<String, Object>> violations = (List<Map<String, Object>>) report.get("violations");
List<String> wcag = violations.stream()
.filter(v -> ((List<String>) v.get("tags")).stream().anyMatch(WCAG_AA::contains))
.map(v -> v.get("id") + " (" + ((List<?>) v.get("nodes")).size() + " nodes)")
.toList();
System.out.println("tb:accessibility: " + violations.size() + " violations, WCAG A/AA: " + wcag);
assertTrue(wcag.isEmpty(), "WCAG violations: " + wcag);
}
On the grid it prints:
tb:accessibility: 3 violations, WCAG A/AA: []
TestingBot session b8676a65ed53-...-6166812 testingbotScanFindsNoWcagViolations(): passed
The report has the same shape as an axe result: violations, passes, incomplete and inapplicable, each rule with an id, tags, impact and nodes. Three violations come back because the command checks best-practice rules too (the same three the Java axe scan found without a tag filter in chapter 3), so the test filters by WCAG tags itself, exactly as withTags did. Locally the test is skipped with an assumption, because the command only exists on the grid.
Use tb:accessibility when you want checks in an existing Selenium suite without adding a dependency, particularly in languages without a maintained axe wrapper. Use the axe wrappers when you need include, exclude or rule selection per scan, or when the same tests must also run on a local browser.
A test asserts that the violations array from tb:accessibility is empty, and it fails on page-has-heading-one. The team's policy is "fail on WCAG A and AA only". What should change?
The report includes best-practice rules such as page-has-heading-one. Each violation carries its tags, so the test can keep only wcag2a, wcag2aa and the other WCAG tags, matching the policy.
Scheduled accessibility scans#
Tests check the states your tests reach. Content editors, marketing pages and third-party scripts change production between deploys, and nobody writes a test for every page. TestingBot's accessibility testing scheduler covers that gap without code:
- Add a URL on the Accessibility page of the member area. The first scan starts right away.
- Set a schedule: once, or recurring daily, weekly, monthly or at a custom interval.
- Choose alerts: email, webhook or SMS.
Each scan checks the page against the axe-core rules for WCAG 2.0 and 2.1 at levels A and AA (axe-core ships the one WCAG 2.2 rule, target-size, switched off), maps each issue to its success criterion, and produces a report with the failing elements and how to fix them. Pages on a local or staging environment can be scanned through TestingBot Tunnel.
The alert rule is designed to avoid the noise of a permanent backlog: you are alerted when a scan finds WCAG A or AA violations the previous scan did not have, or when the page's score drops by more than 5 points. The webhook sends a JSON body such as:
{
"success": false,
"test_id": 203,
"test_name": "Checkout",
"url": "https://example.com/checkout",
"run_id": 49041,
"score": 96,
"wcag_violations": 2,
"needs_review": 112,
"new_violations": ["color-contrast|footer > div > span"],
"report_url": "https://testingbot.com/members/accessibility/203/runs/49041"
}
The fields map onto what you learned in this course: score (0 to 100) counts WCAG A and AA rules only; wcag_violations counts failing elements; needs_review is axe's incomplete, which never fails a scan; and new_violations uses the same rule|selector fingerprint as the baseline in chapter 4. Scans use automated testing minutes from your plan.
A sensible split: pull-request tests for the states your code creates, scheduled scans for the production pages your code does not fully control, and manual audits for what neither can judge.
Testing behind a firewall#
Remote browsers cannot reach localhost or a staging server inside your network. Start TestingBot Tunnel and pass its identifier as tunnelIdentifier in tb:options; the local website testing course explains the tunnel and its options.