Self-Healing Tests: When Healing Hides a Regression

"Self-healing" is the most heavily marketed idea in AI test automation, and the one most worth being careful with. The pitch is simple: the UI changes, a locator stops matching, and instead of a red build the tool finds the new element and carries on. Sometimes that is exactly right. Sometimes the UI change is the bug, and healing turns a regression into a green build.

This chapter breaks a small app on purpose, lets the Playwright healer loose on it twice, and builds the guardrail that keeps a person in the decision.

Two kinds of healing#

Runtime healing happens while the suite runs. When a locator fails, the tool tries alternatives (other attributes, nearby text, visual similarity, a model's guess) and if one matches, the step continues. The test code in your repository does not change. You find out it healed only if you read a report.

Authoring-time healing happens outside the run. An agent sees a failure, proposes an edit to the test file, and the edit goes through review like any other change. The Playwright healer from chapter 3 works this way: it edits files, and nothing reaches your main branch unless someone merges it.

The difference is not the AI. It is who sees the change. Runtime healing makes the decision during the run, before any person looks at it. Authoring-time healing produces a diff that someone can reject. Everything in this chapter assumes the second kind, and the reasons apply even more strongly to the first.

The lab: a shop with two changes#

The one-page shop from chapter 3 starts with a test that passes:

import { test, expect } from '@playwright/test';

test('adds two bags to the cart and shows the total', async ({ page }) => {
  await page.goto('/');
  await page.getByLabel('Quantity').fill('2');
  await page.getByRole('button', { name: 'Add to cart' }).click();
  await expect(page.getByRole('region', { name: 'Cart' }))
    .toContainText('2 x House blend, total 25.00 EUR');
});

Then a release ships two changes at once, which is normal in real life:

1. The button label changes from "Add to cart" to "Add to bag". An intentional copy change.
2. The total calculation stops multiplying by the quantity. A regression.

-    <button id="add" class="btn btn-primary">Add to cart</button>
+    <button id="add" class="btn btn-primary">Add to bag</button>
...
-      const total = (PRICE * qty).toFixed(2);
+      const total = PRICE.toFixed(2);

The test fails on the first change: the click times out because no button is named "Add to cart". A runtime healer would find the button under its new name, continue, and then the assertion would catch the wrong total. So far so good. The real question is what happens when the agent can change assertions too.

Run 1: the healer gets it right#

Use the playwright-test-healer subagent to fix the failing tests in tests/. Report exactly what it changed.

The healer changed one line:

-  await page.getByRole('button', { name: 'Add to cart' }).click();
+  await page.getByRole('button', { name: 'Add to bag' }).click();

and left the test failing, reporting: "Expected: 2 x House blend, total 25.00 EUR. Actual: 2 x House blend, total 12.50 EUR. The cart shows quantity 2 but only the single-bag price, so the app seems not to multiply by quantity." It did not mark the test fixme and did not touch the expected value. That is exactly the behaviour you want: fix the test where the test is stale, report the app where the app is wrong. Cost: $0.18.

Run 2: the healer skips the test#

We restored the total calculation and introduced a different change: the cart now says USD instead of EUR. Is that a bug? It depends on a business decision the page cannot tell you. Same prompt, same healer:

-test('adds two bags to the cart and shows the total', async ({ page }) => {
+test.fixme('adds two bags to the cart and shows the total', async ({ page }) => {
+  // App bug: the cart summary shows "25.00 USD" although the price is listed in EUR
+  // (app/index.html line 22; index.v1/v2.html use EUR). Expected EUR is correct.

The reasoning in the comment is sound, and probably right. But look at what the suite now reports:

  1 skipped
  1 passed (485ms)

Exit code 0. In CI, that is a green build. The currency regression is now recorded only as a code comment in a skipped test. It got there because the healer's instructions tell it to mark a test fixme when it believes the test is correct and the app is wrong. The healer behaved as instructed. The instruction suits a developer at a terminal who reads the summary, and does not suit an unattended pipeline.

Note also that the comment cites index.v1.html and index.v2.html, backup copies we had left in the folder. The agent used them as evidence of the "correct" currency. Agents read whatever is in the workspace, including files you forgot were there.

The guardrail: make skips loud#

Playwright already guards against one kind of silent change: forbidOnly: !!process.env.CI fails the run when someone commits test.only. There is no built-in equivalent for newly added skips, so add one. This script fails when a change adds test.fixme, test.skip or test.only under tests/:

#!/usr/bin/env bash
# Fail the build when a change adds test.fixme / test.skip / test.only.
set -euo pipefail
base="${1:-origin/main}"
added=$(git diff --unified=0 "$base" -- tests | grep -E '^\+.*\btest(\.describe)?\.(fixme|skip|only)\(' || true)
if [ -n "$added" ]; then
  echo "This change disables or focuses tests; a human must approve it:"
  echo "$added"
  exit 1
fi

Run against the healer's change from run 2, it printed the offending line and exited with 1. Wire it into the pull request checks, with a documented way to override it (a label, or a reviewer from a named group), so skipping a test stays possible but always involves a person.

A review rule for healed diffs#

When an agent hands you a healing diff, sort every changed line into one of two buckets:

• Lines that find things: locators, navigation, waits for the app's own signals. These are what healing is for. If the new locator still describes the same thing for the user ("the button that adds to the cart"), approve it.
• Lines that judge things: anything inside expect(...), expected strings and numbers, toHaveCount, skips, removed assertions, loosened regexes. A change here changes the specification. It needs the same justification as a product change: a ticket, a requirement, or a person who owns the behaviour.

In run 1 every changed line was in the first bucket, and the diff could be approved in a minute. Run 2 changed a test's status, which belongs in the second bucket. Most of the risk in healing comes from mixing the two kinds of change in one diff, where the expected-value change looks like a small follow-up to the locator fix.

When healing is the right call#

None of this means "never heal". Healing pays off when:

• the change is known to be intentional (a design refresh, renamed copy, a component library upgrade) and you want 40 locators updated in one pass;
• the healed diff only touches lines that find things;
• someone who knows the feature reviews the diff, and the tests still assert the original requirements afterwards.

Done that way, an agent saves an afternoon of mechanical edits and adds no risk. Done unattended, with power over assertions, it gives you a green build that says nothing about whether the app works.