Exit course
Chapter 9 of 10 Intermediate 9 min read 2 quick checks Updated

A Team Process for Flaky Tests

On this page
  1. Make flakiness visible
  2. Ownership
  3. A flaky budget
  4. A CI policy
  5. Habits in code review
  6. Fix, don't just suppress
  7. Where to go next

Every technique in this course works on one test at a time. A suite of a few thousand tests, written by many people over years, needs something more: a shared agreement about what happens when a test flakes, who looks at it, and how much flakiness the team accepts. Without one, the default process takes over, and the default is "press re-run".

This chapter is about that agreement. It has less code than the others, because the hard part is people and policy, but every piece is something you can put in place this week.

Make flakiness visible#

You cannot manage what nobody sees. Start by producing a list of flaky tests from every CI run. With Playwright's JSON reporter and retries enabled, a short script is enough:

// flaky-report.mjs: list flaky and failed tests from a Playwright JSON report.
import fs from 'node:fs';

const report = JSON.parse(fs.readFileSync(process.argv[2] ?? 'results.json', 'utf8'));
const rows = new Map();
const walk = (suite, file) => {
  for (const spec of suite.specs ?? []) {
    for (const t of spec.tests) {
      const key = `${t.projectName || 'default'} > ${file} > ${spec.title}`;
      const row = rows.get(key) ?? { flaky: 0, failed: 0, runs: 0 };
      row.runs++;
      if (t.status === 'flaky') row.flaky++;
      if (t.status === 'unexpected') row.failed++;
      rows.set(key, row);
    }
  }
  for (const child of suite.suites ?? []) walk(child, file);
};
for (const suite of report.suites) walk(suite, suite.file);

const bad = [...rows].filter(([, r]) => r.flaky || r.failed);
for (const [key, r] of bad) console.log(`${r.flaky} flaky, ${r.failed} failed of ${r.runs}  ${key}`);
process.exitCode = bad.some(([, r]) => r.failed) ? 1 : 0;

On the course suite, with --repeat-each 20 --retries 2, it printed:

1 flaky, 0 failed of 20  default > search.spec.ts > search narrows the results as you type /search
6 flaky, 0 failed of 20  default > search.spec.ts > search shows the right results once it is idle /search
12 flaky, 0 failed of 20  default > timing.spec.ts > save shows a confirmation (flaky)

The run exited with code 0: every test passed in the end. Without the report, three flaky tests (one of them a real product bug) would have been invisible. Post this output on the pull request or in the CI summary.

A single run is a snapshot. Trends need history across runs: the same test, the same environment, over weeks. On TestingBot, the Test Analytics dashboard lists tests that need attention, including flaky ones, and the Insights API (/v1/insights/test-cases?sort=flaky) returns per-test pass rates and a flaky flag you can feed into your own dashboard or a weekly report. Whatever tool you use, the requirement is the same: stable test names and a reported result for every run.

Ownership#

A flaky test without an owner stays flaky. Decide in advance who owns a newly flaky test, so nobody has to negotiate it during an incident:

  • The team that owns the feature owns its tests. A checkout test belongs to the checkout team, not to "QA" or whoever touched it last. A CODEOWNERS file or a tag per test makes this lookup automatic.
  • Whoever introduced it fixes it. If build analytics show that a test started flaking with a specific change, that change's author is the obvious owner.
  • Rotating triage. One person per week reads the flaky list, files issues and assigns them. Triage is not fixing; it is making sure every flaky test has an issue and an owner within a day or two.

Each issue gets the evidence from chapter 2: the command, the pass rate, the error and the environment. "Flaky, please fix" is not an issue; "fails 12 of 20 with an empty status after a 500 ms sleep, Chrome, since build 1042" is.

A flaky budget#

Zero flaky tests is a good aspiration and a poor rule; with a large suite and real browsers, some flakiness will always appear. A budget makes the trade-off explicit, in the same spirit as an error budget for a service. Pick a number the team can measure and agree to act on it, for example:

  • Suite level: at most 1% of CI runs on the main branch need a retry to go green. Measure it weekly.
  • Test level: a test that is flaky in more than, say, 3 of its last 50 runs is quarantined and gets an issue, whatever its history.
  • Time level: no test stays in quarantine longer than two weeks; after that it is fixed, rewritten or deleted.

When the budget is exceeded, the response is decided in advance: feature work on that area pauses until the suite is back under budget, the same way a team that burns its error budget slows down releases. The exact numbers matter less than the commitment to look at them.

The team agreed that at most 1% of main-branch CI runs may need a retry. This month it is 6%. What should the agreed response be?

A budget only works if exceeding it triggers something decided in advance. Moving the number or adding retries makes the metric look better without changing the suite.

A CI policy#

Write the rules down, next to the CI configuration, so they are applied the same way on every pipeline. A policy that works for many teams:

  1. Pull requests: retries on (two at most), with traces on the first retry. Flaky tests are listed in the pull request. A test that is newly added or changed in the pull request and is flaky blocks the merge.
  2. Main branch: --fail-on-flaky-tests (Playwright 1.52+), or the equivalent check in your report script. A flaky result on main creates an issue automatically.
  3. Nightly: the full suite with --shuffle and --repeat-each (or pytest-randomly and --count) on the browsers you support. This is the job that finds order and timing problems before users do. Keep the seed of every failing run.
  4. Quarantine: a tag or marker, a non-blocking job that runs the quarantined tests on every build, and an expiry date in each issue.
  5. Known product bugs: test.fail() or xfail(strict=True) with a link to the bug, never a quarantine.

New tests are the cheapest to fix#

The best time to catch a flaky test is before it merges. When a pull request adds or changes a test, run just that test many times as part of the check:

npx playwright test --only-changed=origin/main --repeat-each 20 --workers 4

--only-changed (Playwright 1.46+) runs the test files changed since the given Git ref, plus test files that import changed files. Playwright's documentation calls the import detection a heuristic, so it complements the full run rather than replacing it. In the course project, committing a branch that changed one test title in timing.spec.ts made it run only that file, 20 times; the flaky save test failed 5 of those runs, which is exactly what this check is for. A new test that cannot pass 20 times in a row on CI is not ready.

Where in the pipeline does running changed tests with --repeat-each 20 give the most value for its cost?

Repeating a handful of new or changed tests is cheap and catches flakiness when the author still has the context. Repeating the whole suite on every pull request multiplies CI time; that belongs in a nightly job.

Habits in code review#

Many flaky tests can be caught by a reviewer who knows what to look for. A short checklist, taken from the previous chapters:

  • A fixed sleep (waitForTimeout, time.sleep, Thread.sleep).
  • A value read from the page and checked with a non-retrying assertion (expect(await locator.textContent()).toBe(...)).
  • A global reset or an "exactly N rows" assertion on shared data.
  • A shared login or a hard-coded account used by more than one test.
  • A test that assumes another test ran first, or sets global state without restoring it.
  • A retry, timeout increase or force: true added "to make it pass" without a measured cause.

Fix, don't just suppress#

Every option in this course falls on a spectrum from suppressing a symptom to removing a cause:

Action Effect
Re-run until green Hides the failure and teaches the team to ignore red
Retry automatically Hides the failure, but records it as flaky if someone looks
Quarantine with an owner Unblocks the team, keeps collecting evidence
Fix the test Removes the cause, if the cause was the test
Fix the app Removes the cause, and a bug users would have hit

Suppression has its place: it keeps CI usable while someone works on the cause. The process above exists to make sure someone always does.

Where to go next#