Running Accessibility Checks in CI
On this page
An accessibility check that only runs when someone remembers to run it decays within weeks. This chapter puts the suite into GitHub Actions so it runs on every push and pull request, keeps the report when something fails, and introduces the policy questions that decide whether the check helps or just annoys: what blocks a merge, who updates the baseline, and how to handle the slow and flaky cases.
The workflow#
Create .github/workflows/accessibility.yml:
name: Accessibility
on:
push:
branches: [main]
pull_request:
jobs:
axe:
runs-on: ubuntu-latest
timeout-minutes: 15
steps:
- uses: actions/checkout@v4
- uses: actions/setup-node@v4
with:
node-version: 22
cache: npm
- run: npm ci
- run: npx playwright install --with-deps chromium
- run: npx playwright test
- uses: actions/upload-artifact@v4
if: ${{ !cancelled() }}
with:
name: playwright-report
path: playwright-report/
retention-days: 14
What each step is for:
-
npm ciinstalls exactly the versions inpackage-lock.json. That matters more for accessibility checks than for most tests: a new axe-core release can add rules or tighten existing ones, and you want that to arrive in a deliberate dependency update, not as a surprise red build on an unrelated pull request. Commit the lock file. -
npx playwright install --with-deps chromiuminstalls only the browser this config uses, plus its Linux libraries. -
npx playwright testruns the suite. GitHub Actions setsCI=true, so the config from chapter 2 switches to the list and HTML reporters, allows one retry and fails on a straytest.only. -
upload-artifactwithif: ${{ !cancelled() }}keeps the HTML report even when tests fail. Every scan attached its full axe JSON to the test (chapter 2), so a failed run carries all the detail needed to fix it, without rerunning anything.
The workflow runs the same commands you run locally. Before pushing, check that the suite passes the way CI will run it:
npm ci
CI=1 npx playwright test
9 passed (2.0s)
and that playwright-report/index.html was written. With a Java suite, the equivalent job uses actions/setup-java@v4 and mvn -B test, and uploads target/surefire-reports.
If you already followed the Playwright tutorial's CI chapter, you can add these specs to the existing job instead of creating a second one. Keeping a separate "Accessibility" job has one advantage: it shows up as its own check on the pull request, which makes it easy to make required on its own.
Monday's builds fail on target-size violations in pages nobody touched. The workflow runs npm install without a lock file. What is the most likely cause?
Without a lock file every run can resolve a new axe-core version. Rule changes are legitimate, but they should arrive in a reviewed dependency update. npm ci with a committed lock file makes the engine version deliberate.
What should block a merge?#
A required check that fails for reasons the author cannot fix gets disabled. Decide the policy up front:
-
New WCAG A and AA violations block the merge. That is what the baseline from chapter 4 enforces. The author introduced the problem, so the author can fix it.
-
Baseline changes need a reviewer who owns accessibility. Put the folder in
CODEOWNERS:/a11y-baseline/ @your-org/accessibilityAdding a line to the baseline is accepting a violation; that decision should not be made alone at 6pm to get a build green.
-
Best-practice rules report but do not block. Run them in a second project or a separate job with
continue-on-error: true, and read them in the report. -
incompleteresults never fail the build. They are questions for a human. Count them in the report, and review them during manual audits.
Where to scan#
Scanning every page in every test makes the suite slow and noisy. A practical split:
- Component or page states that matter: the forms, dialogs, menus and lists your users actually use, as in chapters 4 and 5. These run on every pull request.
- A crawl of the main templates (home, listing, detail, checkout, account), one scan each, also on every pull request. Templates are where a single change breaks hundreds of pages.
- Everything else on a schedule: a nightly job, or a hosted scanner that re-checks your production URLs and alerts on regressions. Chapter 7 covers TestingBot's scheduled accessibility scans for that.
Scans are fast compared with the tests around them: axe typically needs a few hundred milliseconds on a normal page, and the nine-test suite in this course takes about two seconds locally. The cost is in reaching each state, which is why it pays to add scans to tests that already reach those states instead of writing separate navigation for them.
Flakiness and timing#
Accessibility scans are deterministic on a settled page, so flaky scans almost always mean the page was not settled:
-
Scan after a web-first assertion, never directly after an action. In chapter 4,
toHaveText('1 item left')beforeanalyze()guarantees the list has rendered. -
Wait for animations and transitions to finish. Mid-fade text has the wrong contrast. A fade-in that starts at
opacity: 0can produce intermittentcolor-contrastresults. SetreducedMotion: 'reduce'in the Playwrightuseoptions if your CSS honoursprefers-reduced-motion. - Lazy-loaded content only gets checked once it is in the DOM. Scroll to it or wait for it.
- Third-party scripts such as consent banners and chat widgets load at unpredictable times. Either wait for them explicitly or exclude them in the fixture, with a comment saying who owns them.
A dialog scan fails on color-contrast about one run in five, always on the dialog title. What is the most likely cause?
axe measures the colours as rendered at that moment. During a fade the computed opacity is below 1, which lowers contrast. Wait for the dialog's final state (or reduce motion) before scanning.
Making failures readable#
The person reading a failed accessibility check is usually not an accessibility specialist. Help them:
- Keep the short
rule: selectorsassertion message from chapter 2, so the failure fits in the CI log. - Keep the attached axe JSON, which carries
helpUrlfor each rule. - Link to the rule explanations: Deque's rule pages, or TestingBot's accessibility rules overview, which explains each rule and how to fix it.
With the checks running on every pull request in one browser, the next step is to run them on the browsers your users actually have.