Exit course
Chapter 9 of 10 Intermediate 7 min read 2 quick checks Updated

What Automated Accessibility Testing Cannot Catch

On this page
  1. A page that passes and still fails users
  2. What typically needs a human
  3. Pushing automation further: keyboard journeys
  4. Pinning the accessibility tree
  5. Where this leaves you

A green axe scan means "no rule found a problem". It does not mean "accessible", and treating it that way is the most common mistake teams make once their checks are in CI. This chapter shows exactly where the boundary is, with a test that proves it, and then shows how far ordinary Playwright tests can push past it: keyboard journeys and the accessibility tree.

A page that passes and still fails users#

import fs from 'node:fs';
import path from 'node:path';
import AxeBuilder from '@axe-core/playwright';
import { test, expect, WCAG_AA } from './axe';

test('axe passes markup that is still unusable', async ({ page }) => {
  await page.setContent(`<!doctype html><html lang="en"><head><title>Order</title></head><body><main>
    <h1>Your order</h1>
    <img src="data:image/gif;base64,R0lGODlhAQABAAAAACw=" alt="IMG_2041.jpg">
    <input type="text" placeholder="Card number">
    <a href="/invoice.pdf">Click here</a>
  </main></body></html>`);

  const results = await new AxeBuilder({ page }).withTags(WCAG_AA).analyze();

  expect(results.violations).toEqual([]);
});

This test passes in Chrome, Firefox and Safari. Zero WCAG violations. Yet:

  • The image's alt text is a file name. axe checks that alt exists, not that it means anything. A screen reader user hears "IMG_2041.jpg, image" (1.1.1 fails).
  • The card number field is labelled only by a placeholder. axe accepts a placeholder as an accessible name, but the hint disappears as soon as the user types, and many users never see it as a label (3.3.2 Labels or Instructions is at best borderline).
  • The link text is "Click here". In a screen reader's list of links it says nothing about where it goes (2.4.4 Link Purpose depends on context a rules engine cannot judge).

Each of these needs a person to decide whether the text is right for its purpose. That is the general boundary: automation checks that something exists and is technically valid; people check that it is correct and usable.

Every image on a product page has alt="product image", and the axe scan is green. Is the page fine for 1.1.1 Non-text Content?

The image-alt rule verifies presence, not quality. Identical, generic alt text passes the rule but gives a screen reader user no way to tell the products apart.

What typically needs a human#

Use this as a checklist for manual audits, alongside the three passes from chapter 1:

Area Why automation cannot decide
Text alternatives Whether alt text, captions and transcripts are accurate and useful
Link and button wording Whether "Read more" or "Submit" makes sense in context
Focus order Whether the order matches the visual and logical flow
Focus visibility Whether a 1px dotted outline on a busy background is actually visible
Keyboard operation of custom widgets Whether arrow keys, Escape and Enter do the expected thing
Error handling Whether errors are announced, explain the problem and suggest a fix
Dynamic updates Whether toasts, live search results and loading states are announced
Headings and structure Whether headings describe the content they introduce
Timing and motion Whether time limits can be extended and animations stopped
Cognitive load Whether instructions are clear and the flow is understandable
WCAG 2.2 additions Dragging alternatives (2.5.7), consistent help (3.2.6), redundant entry (3.3.7), accessible authentication (3.3.8)

The numbers back this up. When the UK Government Digital Service tested 13 automated tools against a page with 142 deliberate accessibility barriers in 2017, even the best tool found less than half of them. Deque's own analysis of its audit data puts axe-core at about 57% of issues by count, and the share of WCAG success criteria that can be fully decided automatically is far lower than that, because the most common issues (contrast, names, labels) happen to be the automatable ones.

Pushing automation further: keyboard journeys#

You cannot automate judgement, but you can automate a journey once a person has decided what correct looks like. This test completes a todo with the keyboard only, and pins the focus order at every step:

test('keyboard: add and complete a todo without a mouse', async ({ page, browserName }) => {
  // Safari only tabs to text fields unless Option is held (or "Press Tab to highlight
  // each item" is enabled), so WebKit needs Alt+Tab to reach checkboxes and links.
  const tab = browserName === 'webkit' ? 'Alt+Tab' : 'Tab';
  await page.goto('https://demo.playwright.dev/todomvc/');
  const input = page.getByPlaceholder('What needs to be done?');
  await expect(input).toBeFocused(); // the app autofocuses the input

  await page.keyboard.type('Buy milk');
  await page.keyboard.press('Enter');

  await page.keyboard.press(tab);
  await expect(page.getByLabel('Mark all as complete')).toBeFocused();
  await page.keyboard.press(tab);
  const toggle = page.getByRole('checkbox', { name: 'Toggle Todo' });
  await expect(toggle).toBeFocused();
  await page.keyboard.press('Space');
  await expect(toggle).toBeChecked();
});

The first version of this test pressed Tab once and expected the item checkbox. It failed: focus went to "Mark all as complete" first. That is fine behaviour, but it shows why you write these tests after a manual pass, not instead of one: the test records the order a person approved, and fails if a refactor changes it. It also caught the Safari difference from chapter 7.

A keyboard test does not check that focus is visible. Combine it with a screenshot of the focused state if you want to guard against an outline: none regression; the visual regression testing course covers screenshot assertions.

A keyboard test tabs through a form and asserts toBeFocused() on each field. It passes. What can still be broken?

toBeFocused checks document.activeElement, not the focus indicator. A CSS rule removing the outline leaves the test green while making the page unusable for sighted keyboard users (2.4.7).

Pinning the accessibility tree#

Screen readers do not read your HTML; they read the accessibility tree the browser builds from it. Playwright can assert on that tree directly with toMatchAriaSnapshot:

test('sign-up form exposes the right names and roles', async ({ page }) => {
  await page.setContent(fs.readFileSync(path.join(__dirname, 'pages', 'signup-fixed.html'), 'utf8'));

  await expect(page.locator('form')).toMatchAriaSnapshot(`
    - text: Email address
    - textbox "Email address"
    - paragraph: We never share your email.
    - checkbox "Send me product news"
    - text: Send me product news
    - button "Create account"
    - link "Terms"
  `);
});

This is the closest an automated test gets to "what a screen reader announces". If someone replaces the label with a placeholder, the textbox loses its name in the snapshot and the test fails, even though axe would stay green. If the button loses its text, button "Create account" no longer matches. You choose the names, so the snapshot encodes a human decision that axe cannot make.

Keep these snapshots small and focused on components whose names and roles matter, such as forms, navigation and dialogs. A snapshot of a whole page breaks on every content change and teaches people to update it without reading it.

Where this leaves you#

A complete accessibility testing practice has four layers:

  1. axe scans of every important state, on every pull request, with a reviewed baseline (chapters 2 to 6).
  2. Behaviour tests for keyboard journeys and accessible names, written after a manual pass (this chapter).
  3. Scans across browsers and of production pages, on a grid and on a schedule (chapter 7).
  4. Manual audits with a keyboard, a screen reader and zoom, for new features and periodically for the critical journeys (chapter 1), ideally including people who use assistive technology every day.

Skip any one layer and there is a class of bugs nothing catches. The knowledge check in the next chapter tests whether you can tell which layer a given problem belongs to.