AI for Failure Triage: Traces, Logs and Privacy

A red build raises one question before any other: is the app broken, is the test broken, or did the environment get in the way? Answering it means reading an error, a call log, maybe a trace, maybe a server log, and comparing them with what changed. Language models are good at that kind of reading. This chapter covers what to give a model, how to check its answer, and what not to send it.

Playwright already writes the prompt#

When a test fails, Playwright 1.65 writes an error-context.md file next to the test results and prints its path:

Error Context: test-results/demo-fail-free-shipping-above-20-EUR-chromium/error-context.md

Open it and you find a ready-made prompt. This is the start of the file for a failing shop test (the error details sit in a code fence in the real file):

# Instructions

- Following Playwright test failed.
- Explain why, be concise, respect Playwright best practices.
- Provide a snippet of code with the fix, if possible.

# Test info

- Name: demo/fail.spec.ts >> free shipping above 20 EUR
- Location: tests/demo/fail.spec.ts:3:1

# Error details

Error: expect(locator).toContainText(expected) failed

Locator: getByRole('region', { name: 'Cart' })
Timeout: 5000ms
- Expected substring  - 1
+ Received string     + 4

- shipping 0.00 EUR
+
+     Cart
+     2 x House blend, total 25.00 EUR, shipping 4.95 EUR

Below that it includes the call log (what the locator resolved to on each retry) and the page's accessibility snapshot at the moment of failure. The HTML report has a "Copy prompt" button on failed tests that copies the same kind of context. Playwright put it there so that you can paste it into an assistant.

That is a good start, with one catch built into its first lines: "Provide a snippet of code with the fix". The prompt assumes the test needs fixing. For this failure the test is right and the app is wrong: the banner promises free shipping above 20 EUR.

We piped this file, unchanged, into Claude (Sonnet, no tools, $0.05). It did better than the prompt asked. It said the failure was not timing, that either the app was not applying free shipping or the test's expectation was stale, that it could not tell which without the shipping rule, and "Don't loosen the assertion just to make it pass". Its code snippet kept shipping 0.00 EUR. It got there mostly because the test's name, "free shipping above 20 EUR", states the requirement. A test called "cart summary" would have given it nothing to argue with. So the first triage tip costs nothing: name tests after the rule they check.

The model still could not give a verdict, and that is the honest answer. Deciding which side is wrong takes information that is not in the failure.

Ask for a classification, not a fix#

Change the question. Instead of "fix this", ask for a verdict with evidence:

Here is the error context of a failed Playwright test, the diff of the application since the last green run, and the requirement for this feature. Classify the failure as one of: application bug, test bug, environment, flaky timing, unknown. Quote the lines from the error context that support your answer. If the evidence is not enough, say "unknown" and say what you would need to see. Do not propose code changes yet.

Three things make this work better than "explain why":

• The requirement goes in. Without it, the model can only compare the test with the page, and it has no way of knowing which one is wrong. That is the oracle problem from chapter 1 again.
• The diff goes in. "What changed" is most of debugging. A failure right after a commit that touched pricing code is probably a pricing bug.
• Quoted evidence comes out. A model that has to quote the line supporting its claim makes fewer claims it cannot support, and you can check each quote in seconds.

Treat the answer as a hypothesis to check. Models produce confident root causes for ambiguous failures, and timing problems are the hardest case: a single failed run often does not contain enough information to tell a slow backend from a race in the test.

Triage many failures: group first, then ask#

When thirty tests fail overnight, sending thirty error contexts to a model is slow, costly and often pointless, because they usually share two or three causes. Group them deterministically first. This script reads Playwright's JSON report and groups failures by the first line of their error:

// Group failed tests in a Playwright JSON report by the first line of their error.
// Usage: npx playwright test --reporter=json > report.json; node group-failures.mjs report.json
import { readFileSync } from 'node:fs';

const report = JSON.parse(readFileSync(process.argv[2], 'utf8'));
const groups = new Map();

function walk(suite) {
  for (const spec of suite.specs ?? []) {
    for (const t of spec.tests) {
      for (const r of t.results) {
        if (r.status === 'passed' || r.status === 'skipped') continue;
        const key = (r.error?.message ?? r.status).split('\n')[0].replace(/\u001b\[[0-9;]*m/g, '');
        const list = groups.get(key) ?? [];
        list.push(`${spec.file} > ${spec.title} [${t.projectName}] retry ${r.retry}`);
        groups.set(key, list);
      }
    }
  }
  for (const child of suite.suites ?? []) walk(child);
}
report.suites.forEach(walk);

for (const [error, tests] of [...groups].sort((a, b) => b[1].length - a[1].length)) {
  console.log(`${tests.length}x  ${error}`);
  for (const t of tests) console.log(`     ${t}`);
}

On our lab run it printed:

1x  Error: expect(locator).toContainText(expected) failed
     demo/fail.spec.ts > free shipping above 20 EUR [chromium] retry 0

Then send one representative error context per group. The retry number also shows flakiness: a test that failed on retry 0 and passed on retry 1 shows up in Playwright's summary as flaky. For flaky tests, a summary of where in the timeline the failures happen (always the first test after login? only on WebKit? only on the CI runner?) is often more useful than any single error context, and that summary is something a model can draft from the grouped list.

Traces: the richest input, and the most sensitive#

A Playwright trace (trace.zip, opened with npx playwright show-trace) contains a DOM snapshot before and after every action, screenshots, console output and the network log. For a person debugging, it is the best artefact Playwright produces. For a model, it is usually too large to send whole, and it is the artefact most likely to contain things that must not leave your company:

• request and response bodies, including whatever the app returns about the logged-in user;
• cookies and Authorization headers in the network log;
• page content in every DOM snapshot: names, addresses, order histories of test accounts that are sometimes copies of real ones;
• the storageState file from a setup project, which is a working login.

error-context.md is smaller but not clean either: the page snapshot is page text. Before you wire any of this into an automated "send failures to a model" step, decide what the model may see, and strip the rest.

Triage on TestingBot#

If your suite runs on TestingBot (chapter 6), two documented options exist.

AI Insights adds an AI Analysis tab to failed tests. Per its documentation, it sends a curated, text-only slice of the test (failing step, a filtered log tail, status and environment), masks detectable secrets and personal data before sending, and returns a summary, root cause, likely owner (application, test script, environment, flaky or unknown), a confidence score, a timeline, the evidence and suggested fixes. It is off by default, the account owner has to enable it with explicit consent, and the documentation describes the masking as best effort, not anonymisation. It also says it can be wrong, especially on flaky timing. Treat its output like any other model output: a hypothesis to check.

The TestingBot MCP server has a getFailureLogs tool that fetches a session's logs, with a failuresOnly option that keeps only lines matching error-like keywords. In our run on a passing Chrome session, it returned a handful of matched lines that were not errors (a TLS warning, a "status 0" success line), and the output included a signed storage URL for the log file. Keyword filtering is a cheap first pass, not a diagnosis, and the full output is something to keep out of shared chats. Chapter 6 covers the server's other tools.

What to keep human#

Use the model to read faster: summarise a long log, group failures, propose a classification with evidence, draft the bug report. Keep the verdict with a person who knows the requirement, and keep sensitive artefacts out of any provider you have not approved for that data.