Exit course
Chapter 3 of 10 Intermediate 7 min read 2 quick checks Updated

axe-core with Playwright: Your First Scan

On this page
  1. Set up the project
  2. The first scan
  3. Reading the results
  4. Scan the states users see, not just page load
  5. One gotcha: the page needs a browser context
  6. Other frameworks

axe-core is a JavaScript rules engine. It runs inside the page, walks the rendered DOM and the browser's computed styles, and returns a list of rules that passed, failed or could not be decided. @axe-core/playwright is Deque's official wrapper: it injects axe into the page your Playwright test is driving, runs it and hands the results back to your test.

This chapter sets up a project, runs a first scan against TodoMVC and teaches you to read what comes back.

Set up the project#

In an empty folder:

npm init -y
npm i -D @playwright/test @axe-core/playwright
npx playwright install chromium

The course was run with @playwright/test 1.65.0 and @axe-core/playwright 4.14.0, which bundles axe-core 4.14.0. Add a minimal playwright.config.ts:

import { defineConfig, devices } from '@playwright/test';

export default defineConfig({
  testDir: './tests',
  fullyParallel: true,
  forbidOnly: !!process.env.CI,
  retries: process.env.CI ? 1 : 0,
  reporter: process.env.CI ? [['list'], ['html', { open: 'never' }]] : 'list',
  use: {
    trace: 'on-first-retry',
  },
  projects: [
    { name: 'chromium', use: { ...devices['Desktop Chrome'] } },
  ],
});

The first scan#

Create tests/todo.a11y.spec.ts:

import { test, expect } from '@playwright/test';
import AxeBuilder from '@axe-core/playwright';

const WCAG_AA = ['wcag2a', 'wcag2aa', 'wcag21a', 'wcag21aa', 'wcag22a', 'wcag22aa'];

test('todo app has no WCAG A/AA violations on load', async ({ page }, testInfo) => {
  await page.goto('https://demo.playwright.dev/todomvc/');

  const results = await new AxeBuilder({ page }).withTags(WCAG_AA).analyze();

  await testInfo.attach('axe-results.json', {
    body: JSON.stringify(results, null, 2),
    contentType: 'application/json',
  });
  expect(results.violations.map(v => `${v.id}: ${v.nodes.map(n => n.target).join(' | ')}`)).toEqual([]);
});

Three decisions in those few lines:

  • withTags(WCAG_AA) limits the scan to rules that map to WCAG 2.0, 2.1 and 2.2 at levels A and AA. Each version and level has its own tag, so the list needs all six. In axe-core 4.14 no rule carries wcag22a yet (the 2.2 Level A criteria, such as 3.2.6 Consistent Help and 3.3.7 Redundant Entry, need a person to judge), but keeping the tag in the list means future rules for them run without a code change. Without it, axe also runs its best-practice rules, such as "page should have one h1" and "all content should be inside landmarks". They are good advice, but they are not WCAG failures, and mixing them in makes it hard to say what "passing" means. Chapter 4 covers when to add them back.
  • attach stores the full results as JSON on the test, so they appear in the HTML report and you never have to rerun a scan to see the details.
  • The assertion maps violations to short strings before comparing them with an empty array. expect(results.violations).toEqual([]) also works, but on failure it prints hundreds of lines of nested objects. Rule id plus selectors fits on a screen.

Run it:

npx playwright test

It fails:

Error: expect(received).toEqual(expected) // deep equality

- Expected  - 1
+ Received  + 3

- Array []
+ Array [
+   "color-contrast: h1 | p:nth-child(1) | p:nth-child(2) | a[href$=\"remojansen/\"]",
+ ]

That is a real finding on a popular demo: the big pale "todos" heading and the grey footer text do not have enough contrast. Good. A first scan that passes on a real page is more often a sign that the scan is not running where you think it is.

Your scan without withTags fails on region ("all page content should be contained by landmarks"). A colleague says the page fails WCAG. Is that right?

axe tags each rule. Rules tagged best-practice are Deque's recommendations; they improve the experience but are not WCAG failures. Filtering with withTags keeps the two apart so a red build means a WCAG violation.

Reading the results#

analyze() returns an object with four arrays of rules:

  • violations: rules that failed on at least one element. This is what you assert on.
  • passes: rules that ran and passed.
  • incomplete: rules axe could not decide, also called "needs review". Contrast over a background image or a gradient is the usual case; axe cannot compute a reliable ratio, so a human has to look.
  • inapplicable: rules with nothing to check, such as video caption rules on a page without video.

Each violation looks like this (trimmed from the attached JSON):

{
  "id": "color-contrast",
  "impact": "serious",
  "tags": ["cat.color", "wcag2aa", "wcag143", "EN-301-549", "EN-9.1.4.3", "ACT"],
  "help": "Elements must meet minimum color contrast ratio thresholds",
  "helpUrl": "https://dequeuniversity.com/rules/axe/4.14/color-contrast?application=playwright",
  "nodes": [
    {
      "target": ["h1"],
      "failureSummary": "Fix any of the following:\n  Element has insufficient color contrast of 1.26 (foreground color: #ebd7d7, background color: #f5f5f5, font size: 75.0pt (100px), font weight: normal). Expected contrast ratio of 3:1"
    }
  ]
}
  • impact is minor, moderate, serious or critical: axe's estimate of how badly the issue blocks users. It is useful for triage, but a "moderate" violation is still a WCAG failure.
  • tags tell you which criterion failed: wcag143 is 1.4.3 Contrast (Minimum). The EN 301 549 tags map to the European standard.
  • nodes lists every failing element with a CSS selector in target and the reason in failureSummary. Here axe even explains the threshold: large text (100px) needs 3:1, and this heading has 1.26:1.
  • helpUrl links to Deque's explanation of the rule and how to fix it. TestingBot also publishes an overview of the rules it checks, with a page per rule.

Scan the states users see, not just page load#

axe only sees what is in the DOM when analyze() runs. TodoMVC with an empty list has no filter bar, no item checkboxes and no "Clear completed" button, so a load-time scan never checks them.

Add two todos and scan again with the same builder, and the result changes:

color-contrast: h1 | .selected | a[href$="#/active"] | a[href$="#/completed"] | p:nth-child(1) | p:nth-child(2) | a[href$="remojansen/"]

The filter links ("All", "Active", "Completed") appear only once the list has items, and their grey text has a contrast of 4.47:1, just under the 4.5:1 minimum. A load-only scan would never have found them.

The rule of thumb: scan every distinct state a user can reach, typically page load, an open menu or dialog, a form showing validation errors, and a list with data. Write those as normal Playwright tests that drive the UI into the state and then call analyze(). Chapter 4 turns this into a fixture so each scan is one line.

Your checkout page scan is green, but users report that the error messages shown after an invalid submit are unreadable grey. Why did axe miss it?

axe checks the DOM as it is when analyze() runs. Errors that render after a submit only get checked if a test triggers them first. Contrast is a wcag2aa rule, so the tag filter keeps it.

One gotcha: the page needs a browser context#

In Playwright Test the built-in page fixture always works. In a standalone script, though, browser.newPage() makes analyze() throw Error: Please use browser.newContext(). The wrapper opens a helper page in the same context to finish the run, which a page created with the browser.newPage() shortcut does not allow. Create a context first:

const context = await browser.newContext();
const page = await context.newPage();

Other frameworks#

The same engine has official Deque wrappers for other tools: @axe-core/webdriverio (TestingBot's axe and WebdriverIO guide shows the setup on the grid), @axe-core/puppeteer, and Java and .NET packages for Selenium. Cypress users usually reach for the community cypress-axe plugin, which wraps the same engine. The rule ids, tags and result format are the same everywhere, so everything in the following chapters carries over. The next chapter does the same scan from Java.