Retries and Quarantine, Done Right
On this page
Retries and quarantine are tools for keeping CI usable while you fix flaky tests. Used well, they buy time and produce data. Used badly, they turn a test suite into a ritual that always passes and never tells you anything. This chapter measures both effects on the course's flaky tests.
What a retry actually does#
With retries enabled, a failed test runs again, up to the limit. If a later attempt passes, the test counts as passed for the exit code, and the report records that it needed a retry. In Playwright:
// playwright.config.ts
export default defineConfig({
retries: process.env.CI ? 2 : 0,
use: { trace: 'on-first-retry' },
});
Playwright Test reports three outcomes: passed (first attempt), flaky (failed, then passed on a retry) and failed (every attempt failed). trace: 'on-first-retry' records a trace only when a test is retried, which is exactly the run you want to look at.
Each retry runs in a fresh worker process with fresh fixtures, so a retry cannot inherit a broken browser from the failed attempt. testInfo.retry tells a test which attempt it is in, if it needs to clean server-side state before trying again.
Retries hide real bugs#
Here are two of the course's flaky tests, each run 50 times with --retries 2:
npx playwright test tests/timing.spec.ts tests/search.spec.ts -g "flaky|idle /search$" \
--repeat-each 50 --retries 2 --workers 4
| Test | Passed | Flaky | Failed |
|---|---|---|---|
| save with a 500 ms sleep (test problem) | 33 | 16 | 1 |
| search once idle, on the buggy page (app bug) | 36 | 13 | 1 |
Without retries these tests failed about half and a third of the time. With two retries, each failed the build once in 50 runs. The first is a test problem, and the retry hides a slow test. The second is a real race condition in the product, and the retry hides the bug: a user who hits it does not get a retry.
That is the core problem with retries: they cannot tell a flaky test from a flaky product. Both become "flaky" in the report, and if nobody reads that column, both become green.
After enabling retries: 2, the checkout suite has been green for a month. The report shows the same checkout test as flaky in about a third of the runs. What does that tell you?
"Flaky" means the first attempt failed. The retry only changed the exit code. Until someone finds the cause, a third of checkouts might be failing for real users too.
Make flaky visible: fail on flaky, or report it#
Playwright 1.52 added an option that turns flaky results into a failed run:
npx playwright test --retries 2 --fail-on-flaky-tests
On the save test, a run where every test eventually passed but some needed a retry exited with code 1 with the flag and code 0 without it. That is a strong policy: retries still give you a trace of the failure, but nothing flaky merges silently. Many teams use it on the main branch or a nightly run rather than on every pull request.
The softer policy is to keep retries but make the flaky count impossible to ignore: post it in the pull request, track it per test over time, and treat a test that is flaky more than a set number of times as a bug with an owner (chapter 9).
Retries in pytest#
pytest-rerunfailures re-runs failed tests. Restrict reruns to the errors you expect from flakiness, so a genuine assertion about a wrong value is not retried until it happens to pass:
.venv/bin/pytest test_timing.py -k flaky --count 20 -n 4 --reruns 2 --only-rerun AssertionError
4 failed, 16 passed, 18 rerun
--only-rerun takes a regular expression matched against the error, and can be repeated. For a single test, use the marker:
@pytest.mark.flaky(reruns=2, only_rerun=["AssertionError"])
def test_save_with_reruns(driver, base):
...
In a real suite, AssertionError is too broad, because every failed check raises it. The flaky save test happens to fail on an assertion; a better candidate is a TimeoutException from a wait, or a specific infrastructure error such as a dropped connection. The narrower the filter, the less a rerun can hide.
Quarantine: take it out of the blocking path, not out of sight#
A quarantined test still runs, but its result does not block merges. That stops one known flaky test from training everyone to ignore red builds, while the run history keeps collecting evidence. In Playwright, tag it and link the issue:
test('save shows a confirmation (quarantined)', {
tag: '@quarantine',
annotation: { type: 'issue', description: 'https://example.com/issues/123' },
}, async ({ page }) => {
// ...
});
# Blocking job: everything except quarantined tests
npx playwright test --grep-invert @quarantine
# Non-blocking job (allowed to fail): only the quarantined tests, repeated
npx playwright test --grep @quarantine --repeat-each 10
In pytest, register a marker in pytest.ini and select on it:
[pytest]
markers =
quarantine: known flaky, tracked in an issue; excluded from the blocking run
.venv/bin/pytest -m "not quarantine" # blocking
.venv/bin/pytest -m quarantine # non-blocking
Rules that keep quarantine from becoming a graveyard:
- Every quarantined test has an issue and an owner, linked from the test itself.
- Quarantine has an expiry, for example two weeks. After that, the test is fixed, deleted, or escalated.
- The quarantined job still runs on every build and its results are visible, so you know when the fix worked.
- The quarantine list is small. If it keeps growing, the team has a process problem, not a test problem.
Which quarantine setup keeps the most information while unblocking the team?
A skipped or commented-out test produces no more data and is easy to forget. Running it in a non-blocking job keeps its history, so you can see whether it is getting better or worse and confirm the fix.
Known app bugs: expected failures#
When chapter 6's evidence says the product has the bug, do not quarantine the test as flaky. Mark it as an expected failure linked to the bug, using the deterministic version of the test:
test('a slow first response does not win (known bug)', {
annotation: { type: 'issue', description: 'https://example.com/issues/456' },
}, async ({ page }) => {
test.fail(true, 'Stale search results, issue 456');
// ... the deterministic race test from chapter 6
});
Against the buggy page this test is reported as passing, because it failed as expected. Against the fixed page it is reported as failed with "Expected to fail, but passed." That is the right nudge: the day someone fixes the bug, the suite tells them to remove the marker and turn the test into a regression test. In pytest, @pytest.mark.xfail(strict=True, reason="issue 456") behaves the same way.
Where retries are fine#
Retries are reasonable for failures outside your code that you cannot control and do not test: a remote browser that fails to start, a dropped connection to the grid. Even then, prefer retrying the specific operation (starting the session) to retrying the whole test. A rule of thumb:
- Retry infrastructure errors, narrowly, and count them.
- Measure and fix test flakiness; retry it only temporarily, with the flaky count visible.
- Never let retries absorb product failures; mark them as expected failures linked to the bug.
Checklist#
- Retries on CI only, with
trace: 'on-first-retry'so every retried failure leaves evidence. - Flaky results reported and tracked per test, or failed outright with
--fail-on-flaky-testson the branches that matter. - pytest reruns restricted with
--only-rerunto specific errors. - Quarantine via a tag or marker, with an issue, an owner and an expiry, still running in a non-blocking job.
- Known product bugs marked with
test.fail()orxfail(strict=True), never retried away.