--- title: 'Self-Healing Tests: When Healing Hides a Regression' description: ""Self-healing" is the most heavily marketed idea in AI test automation, and the one most worth being careful with. The pitch is simple: the UI…" source_url: html: https://testingbot.com/resources/courses/ai-test-automation/self-healing-tests md: https://testingbot.com/resources/courses/ai-test-automation/self-healing-tests.md --- Course [AI Test Automation: A Hands-On Course](https://testingbot.com/resources/courses/ai-test-automation) Chapter 5 of 9 1. [1 AI Test Automation: A Hands-On Course](https://testingbot.com/resources/courses/ai-test-automation) 2. [2 Set Up an Agent with Playwright MCP](https://testingbot.com/resources/courses/ai-test-automation/set-up-playwright-mcp) 3. [3 From Test Plan to Playwright Test with an Agent](https://testingbot.com/resources/courses/ai-test-automation/generate-tests-from-a-plan) 4. [4 Playwright Test Agents: Planner, Generator and Healer](https://testingbot.com/resources/courses/ai-test-automation/playwright-test-agents) 5. [5 Self-Healing Tests: When Healing Hides a Regression](https://testingbot.com/resources/courses/ai-test-automation/self-healing-tests) 6. [6 AI for Failure Triage: Traces, Logs and Privacy](https://testingbot.com/resources/courses/ai-test-automation/ai-failure-triage) 7. [7 Run AI-Written Suites on TestingBot](https://testingbot.com/resources/courses/ai-test-automation/run-on-testingbot) 8. [8 Guardrails for AI in a Test Suite](https://testingbot.com/resources/courses/ai-test-automation/guardrails) 9. [9 AI Test Automation Quiz](https://testingbot.com/resources/courses/ai-test-automation/knowledge-check) [Exit course](https://testingbot.com/resources/courses) Chapter 5 of 9 Intermediate 8 min read 2 quick checks Updated Nov 2026 # Self-Healing Tests: When Healing Hides a Regression On this page 1. [Two kinds of healing](https://testingbot.com#two-kinds-of-healing) 2. [The lab: a shop with two changes](https://testingbot.com#the-lab-a-shop-with-two-changes) 3. [Run 1: the healer gets it right](https://testingbot.com#run-1-the-healer-gets-it-right) 4. [Run 2: the healer skips the test](https://testingbot.com#run-2-the-healer-skips-the-test) 5. [The guardrail: make skips loud](https://testingbot.com#the-guardrail-make-skips-loud) 6. [A review rule for healed diffs](https://testingbot.com#a-review-rule-for-healed-diffs) 7. [When healing is the right call](https://testingbot.com#when-healing-is-the-right-call) "Self-healing" is the most heavily marketed idea in AI test automation, and the one most worth being careful with. The pitch is simple: the UI changes, a locator stops matching, and instead of a red build the tool finds the new element and carries on. Sometimes that is exactly right. Sometimes the UI change _is_ the bug, and healing turns a regression into a green build. This chapter breaks a small app on purpose, lets the Playwright healer loose on it twice, and builds the guardrail that keeps a person in the decision. ## Two kinds of healing[#](https://testingbot.com#two-kinds-of-healing) **Runtime healing** happens while the suite runs. When a locator fails, the tool tries alternatives (other attributes, nearby text, visual similarity, a model's guess) and if one matches, the step continues. The test code in your repository does not change. You find out it healed only if you read a report. **Authoring-time healing** happens outside the run. An agent sees a failure, proposes an edit to the test file, and the edit goes through review like any other change. The Playwright healer from chapter 3 works this way: it edits files, and nothing reaches your main branch unless someone merges it. The difference is not the AI. It is who sees the change. Runtime healing makes the decision during the run, before any person looks at it. Authoring-time healing produces a diff that someone can reject. Everything in this chapter assumes the second kind, and the reasons apply even more strongly to the first. A runtime self-healing tool reports "healed 14 locators" on last night's green run. What is the risk? None; the tests passed, so the application works Some of those heals may have matched a different element or hidden a real UI change, and nobody reviewed them The run will be slower next time The locators will heal back to the old version tomorrow Each heal is a decision that "this other element is the one I meant". Some will be right. But a removed button, a moved price or a renamed legal checkbox can also be "healed" into a pass. A heal count on a green run is a list of unreviewed changes, not a measure of quality. ## The lab: a shop with two changes[#](https://testingbot.com#the-lab-a-shop-with-two-changes) The one-page shop from chapter 3 starts with a test that passes: ```typescript import { test, expect } from '@playwright/test'; test('adds two bags to the cart and shows the total', async ({ page }) => { await page.goto('/'); await page.getByLabel('Quantity').fill('2'); await page.getByRole('button', { name: 'Add to cart' }).click(); await expect(page.getByRole('region', { name: 'Cart' })) .toContainText('2 x House blend, total 25.00 EUR'); }); ``` Then a release ships two changes at once, which is normal in real life: 1. The button label changes from "Add to cart" to "Add to bag". An intentional copy change. 2. The total calculation stops multiplying by the quantity. A regression. ```diff - + ... - const total = (PRICE * qty).toFixed(2); + const total = PRICE.toFixed(2); ``` The test fails on the first change: the click times out because no button is named "Add to cart". A runtime healer would find the button under its new name, continue, and then the assertion would catch the wrong total. So far so good. The real question is what happens when the agent can change assertions too. ## Run 1: the healer gets it right[#](https://testingbot.com#run-1-the-healer-gets-it-right) > Use the playwright-test-healer subagent to fix the failing tests in tests/. Report exactly what it changed. The healer changed one line: ```diff - await page.getByRole('button', { name: 'Add to cart' }).click(); + await page.getByRole('button', { name: 'Add to bag' }).click(); ``` and left the test failing, reporting: "Expected: `2 x House blend, total 25.00 EUR`. Actual: `2 x House blend, total 12.50 EUR`. The cart shows quantity 2 but only the single-bag price, so the app seems not to multiply by quantity." It did not mark the test `fixme` and did not touch the expected value. That is exactly the behaviour you want: fix the test where the test is stale, report the app where the app is wrong. Cost: $0.18. ## Run 2: the healer skips the test[#](https://testingbot.com#run-2-the-healer-skips-the-test) We restored the total calculation and introduced a different change: the cart now says `USD` instead of `EUR`. Is that a bug? It depends on a business decision the page cannot tell you. Same prompt, same healer: ```diff -test('adds two bags to the cart and shows the total', async ({ page }) => { +test.fixme('adds two bags to the cart and shows the total', async ({ page }) => { + // App bug: the cart summary shows "25.00 USD" although the price is listed in EUR + // (app/index.html line 22; index.v1/v2.html use EUR). Expected EUR is correct. ``` The reasoning in the comment is sound, and probably right. But look at what the suite now reports: ``` 1 skipped 1 passed (485ms) ``` Exit code 0. In CI, that is a green build. The currency regression is now recorded only as a code comment in a skipped test. It got there because the healer's instructions tell it to mark a test `fixme` when it believes the test is correct and the app is wrong. The healer behaved as instructed. The instruction suits a developer at a terminal who reads the summary, and does not suit an unattended pipeline. Note also that the comment cites `index.v1.html` and `index.v2.html`, backup copies we had left in the folder. The agent used them as evidence of the "correct" currency. Agents read whatever is in the workspace, including files you forgot were there. A healer run in CI changes a test from `test(...)` to `test.fixme(...)` with a comment explaining an app bug. The pipeline is green. What is the correct policy? Accept it automatically; the comment documents the bug Revert it automatically and rerun the healer until it produces a passing test Allow it only on weekends, when fewer people deploy Fail the pipeline on any newly added skip or fixme, so a person decides whether the app or the test is wrong Skipping a test is a decision about whether a behaviour is acceptable, and that needs a person. Rerunning the healer until it passes is worse: the easiest way to make it pass is to change the expected value to match the bug. ## The guardrail: make skips loud[#](https://testingbot.com#the-guardrail-make-skips-loud) Playwright already guards against one kind of silent change: `forbidOnly: !!process.env.CI` fails the run when someone commits `test.only`. There is no built-in equivalent for newly added skips, so add one. This script fails when a change adds `test.fixme`, `test.skip` or `test.only` under `tests/`: ```bash #!/usr/bin/env bash # Fail the build when a change adds test.fixme / test.skip / test.only. set -euo pipefail base="${1:-origin/main}" added=$(git diff --unified=0 "$base" -- tests | grep -E '^\+.*\btest(\.describe)?\.(fixme|skip|only)\(' || true) if [-n "$added"]; then echo "This change disables or focuses tests; a human must approve it:" echo "$added" exit 1 fi ``` Run against the healer's change from run 2, it printed the offending line and exited with 1. Wire it into the pull request checks, with a documented way to override it (a label, or a reviewer from a named group), so skipping a test stays possible but always involves a person. ## A review rule for healed diffs[#](https://testingbot.com#a-review-rule-for-healed-diffs) When an agent hands you a healing diff, sort every changed line into one of two buckets: - **Lines that find things** : locators, navigation, waits for the app's own signals. These are what healing is for. If the new locator still describes the same thing for the user ("the button that adds to the cart"), approve it. - **Lines that judge things** : anything inside `expect(...)`, expected strings and numbers, `toHaveCount`, skips, removed assertions, loosened regexes. A change here changes the specification. It needs the same justification as a product change: a ticket, a requirement, or a person who owns the behaviour. In run 1 every changed line was in the first bucket, and the diff could be approved in a minute. Run 2 changed a test's status, which belongs in the second bucket. Most of the risk in healing comes from mixing the two kinds of change in one diff, where the expected-value change looks like a small follow-up to the locator fix. ## When healing is the right call[#](https://testingbot.com#when-healing-is-the-right-call) None of this means "never heal". Healing pays off when: - the change is known to be intentional (a design refresh, renamed copy, a component library upgrade) and you want 40 locators updated in one pass; - the healed diff only touches lines that find things; - someone who knows the feature reviews the diff, and the tests still assert the original requirements afterwards. Done that way, an agent saves an afternoon of mechanical edits and adds no risk. Done unattended, with power over assertions, it gives you a green build that says nothing about whether the app works. [Previous · Chapter 4 Playwright Test Agents: Planner, Generator and Healer](https://testingbot.com/resources/courses/ai-test-automation/playwright-test-agents) [Next · Chapter 6 AI for Failure Triage: Traces, Logs and Privacy](https://testingbot.com/resources/courses/ai-test-automation/ai-failure-triage) On this page 1. [Two kinds of healing](https://testingbot.com#two-kinds-of-healing) 2. [The lab: a shop with two changes](https://testingbot.com#the-lab-a-shop-with-two-changes) 3. [Run 1: the healer gets it right](https://testingbot.com#run-1-the-healer-gets-it-right) 4. [Run 2: the healer skips the test](https://testingbot.com#run-2-the-healer-skips-the-test) 5. [The guardrail: make skips loud](https://testingbot.com#the-guardrail-make-skips-loud) 6. [A review rule for healed diffs](https://testingbot.com#a-review-rule-for-healed-diffs) 7. [When healing is the right call](https://testingbot.com#when-healing-is-the-right-call) Certification This course has a free, hands-on certificate. [How it works](https://testingbot.com/resources/courses/ai-test-automation/certification)