---
title: Retries and Quarantine, Done Right | TestingBot
description: Retries and quarantine are tools for keeping CI usable while you fix
  flaky tests. Used well, they buy time and produce data. Used badly, they turn a
  test…
source_url:
  html: https://testingbot.com/resources/courses/flaky-tests/retries-and-quarantine
  md: https://testingbot.com/resources/courses/flaky-tests/retries-and-quarantine.md
---

Course

[Flaky Tests: Find, Fix and Prevent](https://testingbot.com/resources/courses/flaky-tests)

Chapter 7 of 10

1. [1 Flaky Tests: How to Find, Fix and Prevent Them](https://testingbot.com/resources/courses/flaky-tests)
2. [2 Measure Flakiness Before You Fix It](https://testingbot.com/resources/courses/flaky-tests/measuring-flakiness)
3. [3 Root Cause 1: Timing, Sleeps and Waits](https://testingbot.com/resources/courses/flaky-tests/timing-and-waits)
4. [4 Root Cause 2: Test Data and Shared State](https://testingbot.com/resources/courses/flaky-tests/test-data-and-shared-state)
5. [5 Root Cause 3: Test Order and Isolation](https://testingbot.com/resources/courses/flaky-tests/test-order-and-isolation)
6. [6 Root Cause 4: When the App Is the Flaky Part](https://testingbot.com/resources/courses/flaky-tests/when-the-app-is-flaky)
7. [7 Retries and Quarantine, Done Right](https://testingbot.com/resources/courses/flaky-tests/retries-and-quarantine)
8. [8 Debugging Flaky Tests on a Cloud Grid](https://testingbot.com/resources/courses/flaky-tests/debugging-flaky-tests-on-a-cloud-grid)
9. [9 A Team Process for Flaky Tests](https://testingbot.com/resources/courses/flaky-tests/a-team-process-for-flaky-tests)
10. [10 Flaky Tests Knowledge Check](https://testingbot.com/resources/courses/flaky-tests/knowledge-check)

[Exit course](https://testingbot.com/resources/courses)

 Chapter 7 of 10 Intermediate 8 min read 2 quick checks Updated Nov 2026 

# Retries and Quarantine, Done Right
On this page
1. [What a retry actually does](https://testingbot.com#what-a-retry-actually-does)
2. [Retries hide real bugs](https://testingbot.com#retries-hide-real-bugs)
3. [Make flaky visible: fail on flaky, or report it](https://testingbot.com#make-flaky-visible-fail-on-flaky-or-report-it)
4. [Retries in pytest](https://testingbot.com#retries-in-pytest)
5. [Quarantine: take it out of the blocking path, not out of sight](https://testingbot.com#quarantine-take-it-out-of-the-blocking-path-not-out-of-sight)
6. [Known app bugs: expected failures](https://testingbot.com#known-app-bugs-expected-failures)
7. [Where retries are fine](https://testingbot.com#where-retries-are-fine)
8. [Checklist](https://testingbot.com#checklist)

Retries and quarantine are tools for keeping CI usable while you fix flaky tests. Used well, they buy time and produce data. Used badly, they turn a test suite into a ritual that always passes and never tells you anything. This chapter measures both effects on the course's flaky tests.

## What a retry actually does[#](https://testingbot.com#what-a-retry-actually-does)

With retries enabled, a failed test runs again, up to the limit. If a later attempt passes, the test counts as passed for the exit code, and the report records that it needed a retry. In Playwright:

```typescript
// playwright.config.ts
export default defineConfig({
  retries: process.env.CI ? 2 : 0,
  use: { trace: 'on-first-retry' },
});
```

Playwright Test reports three outcomes: **passed** (first attempt), **flaky** (failed, then passed on a retry) and **failed** (every attempt failed). `trace: 'on-first-retry'` records a trace only when a test is retried, which is exactly the run you want to look at.

Each retry runs in a fresh worker process with fresh fixtures, so a retry cannot inherit a broken browser from the failed attempt. `testInfo.retry` tells a test which attempt it is in, if it needs to clean server-side state before trying again.

## Retries hide real bugs[#](https://testingbot.com#retries-hide-real-bugs)

Here are two of the course's flaky tests, each run 50 times with `--retries 2`:

```bash
npx playwright test tests/timing.spec.ts tests/search.spec.ts -g "flaky|idle /search$" \
  --repeat-each 50 --retries 2 --workers 4
```

| Test | Passed | Flaky | Failed |
| --- | --- | --- | --- |
| save with a 500 ms sleep (test problem) | 33 | 16 | 1 |
| search once idle, on the buggy page (app bug) | 36 | 13 | 1 |

Without retries these tests failed about half and a third of the time. With two retries, each failed the build once in 50 runs. The first is a test problem, and the retry hides a slow test. The second is a real race condition in the product, and the retry hides the bug: a user who hits it does not get a retry.

That is the core problem with retries: they cannot tell a flaky test from a flaky product. Both become "flaky" in the report, and if nobody reads that column, both become green.

After enabling `retries: 2`, the checkout suite has been green for a month. The report shows the same checkout test as flaky in about a third of the runs. What does that tell you?

 The retries fixed the test The test still fails about a third of the time; the cause, in the test or in checkout, is unknown and still there The test is fine; flaky only means it was slow

"Flaky" means the first attempt failed. The retry only changed the exit code. Until someone finds the cause, a third of checkouts might be failing for real users too.

## Make flaky visible: fail on flaky, or report it[#](https://testingbot.com#make-flaky-visible-fail-on-flaky-or-report-it)

Playwright 1.52 added an option that turns flaky results into a failed run:

```bash
npx playwright test --retries 2 --fail-on-flaky-tests
```

On the save test, a run where every test eventually passed but some needed a retry exited with code 1 with the flag and code 0 without it. That is a strong policy: retries still give you a trace of the failure, but nothing flaky merges silently. Many teams use it on the main branch or a nightly run rather than on every pull request.

The softer policy is to keep retries but make the flaky count impossible to ignore: post it in the pull request, track it per test over time, and treat a test that is flaky more than a set number of times as a bug with an owner (chapter 9).

## Retries in pytest[#](https://testingbot.com#retries-in-pytest)

pytest-rerunfailures re-runs failed tests. Restrict reruns to the errors you expect from flakiness, so a genuine assertion about a wrong value is not retried until it happens to pass:

```bash
.venv/bin/pytest test_timing.py -k flaky --count 20 -n 4 --reruns 2 --only-rerun AssertionError
```

```
4 failed, 16 passed, 18 rerun
```

`--only-rerun` takes a regular expression matched against the error, and can be repeated. For a single test, use the marker:

```python
@pytest.mark.flaky(reruns=2, only_rerun=["AssertionError"])
def test_save_with_reruns(driver, base):
    ...
```

In a real suite, `AssertionError` is too broad, because every failed check raises it. The flaky save test happens to fail on an assertion; a better candidate is a `TimeoutException` from a wait, or a specific infrastructure error such as a dropped connection. The narrower the filter, the less a rerun can hide.

## Quarantine: take it out of the blocking path, not out of sight[#](https://testingbot.com#quarantine-take-it-out-of-the-blocking-path-not-out-of-sight)

A quarantined test still runs, but its result does not block merges. That stops one known flaky test from training everyone to ignore red builds, while the run history keeps collecting evidence. In Playwright, tag it and link the issue:

```typescript
test('save shows a confirmation (quarantined)', {
  tag: '@quarantine',
  annotation: { type: 'issue', description: 'https://example.com/issues/123' },
}, async ({ page }) => {
  // ...
});
```

```bash
# Blocking job: everything except quarantined tests
npx playwright test --grep-invert @quarantine

# Non-blocking job (allowed to fail): only the quarantined tests, repeated
npx playwright test --grep @quarantine --repeat-each 10
```

In pytest, register a marker in `pytest.ini` and select on it:

```ini
[pytest]
markers =
    quarantine: known flaky, tracked in an issue; excluded from the blocking run
```

```bash
.venv/bin/pytest -m "not quarantine" # blocking
.venv/bin/pytest -m quarantine # non-blocking
```

Rules that keep quarantine from becoming a graveyard:

- Every quarantined test has an issue and an owner, linked from the test itself.
- Quarantine has an expiry, for example two weeks. After that, the test is fixed, deleted, or escalated.
- The quarantined job still runs on every build and its results are visible, so you know when the fix worked.
- The quarantine list is small. If it keeps growing, the team has a process problem, not a test problem.

Which quarantine setup keeps the most information while unblocking the team?

 The test is tagged, excluded from the blocking job, still run in a non-blocking job, and linked to an issue with an owner The test is skipped with `test.skip()` until someone has time The test is commented out and a TODO is added The test gets `retries: 10`

A skipped or commented-out test produces no more data and is easy to forget. Running it in a non-blocking job keeps its history, so you can see whether it is getting better or worse and confirm the fix.

## Known app bugs: expected failures[#](https://testingbot.com#known-app-bugs-expected-failures)

When chapter 6's evidence says the product has the bug, do not quarantine the test as flaky. Mark it as an expected failure linked to the bug, using the deterministic version of the test:

```typescript
test('a slow first response does not win (known bug)', {
  annotation: { type: 'issue', description: 'https://example.com/issues/456' },
}, async ({ page }) => {
  test.fail(true, 'Stale search results, issue 456');
  // ... the deterministic race test from chapter 6
});
```

Against the buggy page this test is reported as passing, because it failed as expected. Against the fixed page it is reported as **failed** with "Expected to fail, but passed." That is the right nudge: the day someone fixes the bug, the suite tells them to remove the marker and turn the test into a regression test. In pytest, `@pytest.mark.xfail(strict=True, reason="issue 456")` behaves the same way.

## Where retries are fine[#](https://testingbot.com#where-retries-are-fine)

Retries are reasonable for failures outside your code that you cannot control and do not test: a remote browser that fails to start, a dropped connection to the grid. Even then, prefer retrying the specific operation (starting the session) to retrying the whole test. A rule of thumb:

- Retry **infrastructure** errors, narrowly, and count them.
- Measure and fix **test** flakiness; retry it only temporarily, with the flaky count visible.
- Never let retries absorb **product** failures; mark them as expected failures linked to the bug.

## Checklist[#](https://testingbot.com#checklist)

- Retries on CI only, with `trace: 'on-first-retry'` so every retried failure leaves evidence.
- Flaky results reported and tracked per test, or failed outright with `--fail-on-flaky-tests` on the branches that matter.
- pytest reruns restricted with `--only-rerun` to specific errors.
- Quarantine via a tag or marker, with an issue, an owner and an expiry, still running in a non-blocking job.
- Known product bugs marked with `test.fail()` or `xfail(strict=True)`, never retried away.

[Previous · Chapter 6 Root Cause 4: When the App Is the Flaky Part](https://testingbot.com/resources/courses/flaky-tests/when-the-app-is-flaky) [Next · Chapter 8 Debugging Flaky Tests on a Cloud Grid](https://testingbot.com/resources/courses/flaky-tests/debugging-flaky-tests-on-a-cloud-grid)

On this page

1. [What a retry actually does](https://testingbot.com#what-a-retry-actually-does)
2. [Retries hide real bugs](https://testingbot.com#retries-hide-real-bugs)
3. [Make flaky visible: fail on flaky, or report it](https://testingbot.com#make-flaky-visible-fail-on-flaky-or-report-it)
4. [Retries in pytest](https://testingbot.com#retries-in-pytest)
5. [Quarantine: take it out of the blocking path, not out of sight](https://testingbot.com#quarantine-take-it-out-of-the-blocking-path-not-out-of-sight)
6. [Known app bugs: expected failures](https://testingbot.com#known-app-bugs-expected-failures)
7. [Where retries are fine](https://testingbot.com#where-retries-are-fine)
8. [Checklist](https://testingbot.com#checklist)

Certification

Earn the free, resume-worthy TestingBot Flaky Test Fixing Certification for this course.

[How it works](https://testingbot.com/resources/courses/flaky-tests/certification)
