Scoping Scans, Rules and a Baseline of Known Issues
On this page
The first scan of a real application almost never passes. That leaves you with an awkward choice: fix everything before the check can go into CI, which can take months, or leave the check out of CI, which means new violations pile on top of the old ones. Neither is good.
This chapter shows the way out: decide what a scan covers, configure the rules on purpose rather than by accident, and record today's violations in a baseline so the build fails only on new ones while the backlog gets fixed.
A shared fixture#
Every scan in a suite should use the same tags and exclusions, so put the builder in a Playwright fixture. Create tests/axe.ts:
import { test as base } from '@playwright/test';
import AxeBuilder from '@axe-core/playwright';
export const WCAG_AA = ['wcag2a', 'wcag2aa', 'wcag21a', 'wcag21aa', 'wcag22a', 'wcag22aa'];
type AxeFixtures = { makeAxeBuilder: () => AxeBuilder };
export const test = base.extend<AxeFixtures>({
makeAxeBuilder: async ({ page }, use) => {
await use(() =>
new AxeBuilder({ page })
.withTags(WCAG_AA)
// Third-party credits footer we do not own; tracked with its maintainers instead.
.exclude('footer.info'),
);
},
});
export { expect } from '@playwright/test';
The fixture returns a function rather than a builder, so each call gets a fresh AxeBuilder and a test can still add its own include or disableRules on top. Tests now import test and expect from ./axe instead of @playwright/test. The single test from chapter 2 is replaced by the specs below, so you can delete tests/todo.a11y.spec.ts.
Scoping: include and exclude#
include(selector) limits the scan to part of the page; exclude(selector) removes part of it. Both take CSS selectors and can be chained.
Exclude content you do not control and cannot change, and write down why next to the selector, as the fixture does. Typical candidates are embedded third-party widgets such as video players, maps and chat bubbles, which is exactly where the Mars demo's ARIA violations came from in chapter 3. Excluding them does not make them accessible; it moves them from "red build" to "a vendor issue someone owns". If nobody owns it, do not exclude it.
Include is useful for component-level tests. To check just TodoMVC's filter bar:
test('filter bar, scanned on its own', async ({ page }) => {
const input = page.getByPlaceholder('What needs to be done?');
await input.fill('Buy milk');
await input.press('Enter');
const results = await new AxeBuilder({ page })
.include('.filters')
.disableRules(['color-contrast'])
.analyze();
expect(results.violations).toEqual([]);
});
This test also shows the riskiest option: disableRules.
Choosing rules on purpose#
You have four levers:
| Method | Effect |
|---|---|
withTags([...]) |
Run only rules that carry at least one of these tags |
withRules([...]) |
Run only these rule ids |
disableRules([...]) |
Run everything selected, except these rule ids |
options({...}) |
Pass raw axe-core run options, for anything the helpers do not cover |
Useful tags beyond the WCAG ones: best-practice for Deque's recommendations (heading order, one h1, landmarks), wcag2aaa for the few AAA rules axe has, such as enhanced contrast, and experimental for rules still being tuned. A sensible policy is to fail the build on WCAG A and AA, and to run best-practice in a separate, non-blocking report.
disableRules switches a check off everywhere the builder is used. Disabling color-contrast in the component test above is defensible only because contrast is covered by the full-page scans; disabling it in the shared fixture would hide every contrast bug in the product. Treat disableRules in a shared fixture like a skipped test: rare, commented and reviewed. For a single known issue, a baseline is a better tool, because it stays specific to the element that is broken.
The header logo fails color-contrast and the designer needs two sprints to fix it. Which change keeps the build useful in the meantime?
A baseline accepts exactly the known element while still failing on any new contrast problem. Disabling the rule or the whole AA tag would hide every future regression of that kind, and fixme stops checking the page altogether.
A baseline of known violations#
A baseline is a committed file listing the violations you already know about. The test compares each scan against it and fails on anything new. Create tests/a11y-baseline.ts:
import fs from 'node:fs';
import path from 'node:path';
import type { AxeResults } from 'axe-core';
const DIR = path.join(__dirname, '..', 'a11y-baseline');
// One line per failing element: "rule|selector". Stable, readable in a diff.
export function fingerprints(results: AxeResults): string[] {
return results.violations
.flatMap(v => v.nodes.map(n => `${v.id}|${n.target.join(' ')}`))
.sort();
}
export function compareWithBaseline(name: string, results: AxeResults) {
const file = path.join(DIR, `${name}.json`);
const current = fingerprints(results);
if (process.env.UPDATE_A11Y_BASELINE) {
fs.mkdirSync(DIR, { recursive: true });
fs.writeFileSync(file, JSON.stringify(current, null, 2) + '\n');
return { newIssues: [], fixedIssues: [] };
}
const known: string[] = fs.existsSync(file) ? JSON.parse(fs.readFileSync(file, 'utf8')) : [];
return {
newIssues: current.filter(f => !known.includes(f)),
fixedIssues: known.filter(f => !current.includes(f)),
};
}
(axe-core is installed as a dependency of @axe-core/playwright, so the type import works without installing anything else.)
Design choices worth keeping if you write your own:
-
The fingerprint is rule plus selector, not the failure message. Messages contain measured values such as contrast ratios that change with every tweak; the rule and element are what you are tracking. TestingBot's scheduled scans use the same
rule|selectoridea to decide which violations are new. - One file per scanned state, so parallel tests never write the same file, and a diff shows which page changed.
- Fixed issues fail too. If an entry disappears from the scan, the test asks you to remove it from the baseline. Otherwise the file only ever grows, and a fixed bug can quietly come back later without failing anything.
Now the spec, tests/todo.baseline.spec.ts, scanning two states:
import AxeBuilder from '@axe-core/playwright';
import { test, expect } from './axe';
import { compareWithBaseline } from './a11y-baseline';
test.beforeEach(async ({ page }) => {
await page.goto('https://demo.playwright.dev/todomvc/');
});
test('empty list', async ({ makeAxeBuilder }) => {
const results = await makeAxeBuilder().analyze();
const { newIssues, fixedIssues } = compareWithBaseline('todo-empty', results);
expect(newIssues, 'new accessibility violations').toEqual([]);
expect(fixedIssues, 'fixed, so remove them from the baseline').toEqual([]);
});
test('list with an active and a completed item', async ({ page, makeAxeBuilder }) => {
const input = page.getByPlaceholder('What needs to be done?');
await input.fill('Buy milk');
await input.press('Enter');
await input.fill('Pay rent');
await input.press('Enter');
await page.getByRole('listitem').filter({ hasText: 'Pay rent' }).getByRole('checkbox').check();
await expect(page.getByTestId('todo-count')).toHaveText('1 item left');
const results = await makeAxeBuilder().analyze();
const { newIssues, fixedIssues } = compareWithBaseline('todo-with-items', results);
expect(newIssues, 'new accessibility violations').toEqual([]);
expect(fixedIssues, 'fixed, so remove them from the baseline').toEqual([]);
});
The filter bar test from earlier goes in the same file. Note the toHaveText('1 item left') assertion before the second scan: it waits until the UI has settled, so axe does not scan a half-updated page.
The first run without a baseline fails and lists every violation as new. Review that list, then record it:
UPDATE_A11Y_BASELINE=1 npx playwright test tests/todo.baseline.spec.ts
npx playwright test tests/todo.baseline.spec.ts
The second run passes. a11y-baseline/todo-with-items.json now reads:
[
"color-contrast|.selected",
"color-contrast|a[href$=\"#/active\"]",
"color-contrast|a[href$=\"#/completed\"]",
"color-contrast|h1"
]
The footer text from chapter 2 is gone because the fixture excludes footer.info. The heading and filter links are real, owned problems, so they are in the baseline, not excluded.
A developer fixed the filter link contrast and the baseline test now fails with "fixed, so remove them from the baseline". Why fail on good news?
A baseline entry is permission for that violation to exist. Once it is fixed, keeping the permission means the bug can return without failing the build. Updating the file locks in the improvement.
Keeping the baseline honest#
A baseline is debt, so manage it like debt:
-
Review baseline diffs like code. A pull request that adds lines to
a11y-baseline/is accepting new violations. Make it visible: require a code owner on the folder, and never regenerate the baseline as a reflex to get a build green. - Track the count. The number of lines across the baseline files is a simple metric that should only go down.
- Expect selector churn. axe generates the shortest unique selector it can, so a markup refactor can change a selector and turn a known issue into a "new" one. Read the diff before you regenerate; if the rule and element are the same, updating is correct.
- Re-check the baseline per browser if you scan several. The next chapter and chapter 7 show that WebKit can report a violation Chromium does not.
You now have scans that are scoped, configured on purpose and safe to put in front of every pull request. Before doing that, the next chapter fixes the violations themselves.