---
title: Measure Flakiness Before You Fix It | TestingBot
description: A flaky test fails some of the time. Before you change anything, find
  out how often, under which conditions, and with which error. That number is what
  tells…
source_url:
  html: https://testingbot.com/resources/courses/flaky-tests/measuring-flakiness
  md: https://testingbot.com/resources/courses/flaky-tests/measuring-flakiness.md
---

Course

[Flaky Tests: Find, Fix and Prevent](https://testingbot.com/resources/courses/flaky-tests)

Chapter 2 of 10

1. [1 Flaky Tests: How to Find, Fix and Prevent Them](https://testingbot.com/resources/courses/flaky-tests)
2. [2 Measure Flakiness Before You Fix It](https://testingbot.com/resources/courses/flaky-tests/measuring-flakiness)
3. [3 Root Cause 1: Timing, Sleeps and Waits](https://testingbot.com/resources/courses/flaky-tests/timing-and-waits)
4. [4 Root Cause 2: Test Data and Shared State](https://testingbot.com/resources/courses/flaky-tests/test-data-and-shared-state)
5. [5 Root Cause 3: Test Order and Isolation](https://testingbot.com/resources/courses/flaky-tests/test-order-and-isolation)
6. [6 Root Cause 4: When the App Is the Flaky Part](https://testingbot.com/resources/courses/flaky-tests/when-the-app-is-flaky)
7. [7 Retries and Quarantine, Done Right](https://testingbot.com/resources/courses/flaky-tests/retries-and-quarantine)
8. [8 Debugging Flaky Tests on a Cloud Grid](https://testingbot.com/resources/courses/flaky-tests/debugging-flaky-tests-on-a-cloud-grid)
9. [9 A Team Process for Flaky Tests](https://testingbot.com/resources/courses/flaky-tests/a-team-process-for-flaky-tests)
10. [10 Flaky Tests Knowledge Check](https://testingbot.com/resources/courses/flaky-tests/knowledge-check)

[Exit course](https://testingbot.com/resources/courses)

 Chapter 2 of 10 Intermediate 10 min read 2 quick checks Updated Nov 2026 

# Measure Flakiness Before You Fix It
On this page
1. [The demo app](https://testingbot.com#the-demo-app)
2. [The Playwright project](https://testingbot.com#the-playwright-project)
3. [The Python project](https://testingbot.com#the-python-project)
4. [Run one test many times](https://testingbot.com#run-one-test-many-times)
5. [How many runs are enough](https://testingbot.com#how-many-runs-are-enough)
6. [Expose order and parallel problems on purpose](https://testingbot.com#expose-order-and-parallel-problems-on-purpose)
7. [What retries tell you](https://testingbot.com#what-retries-tell-you)
8. [History beats a single run](https://testingbot.com#history-beats-a-single-run)
9. [A measurement log](https://testingbot.com#a-measurement-log)

A flaky test fails some of the time. Before you change anything, find out how often, under which conditions, and with which error. That number is what tells you later whether your fix worked. This chapter sets up the demo app and the two test projects, then shows the measuring tools used in every later chapter.

## The demo app[#](https://testingbot.com#the-demo-app)

Create a folder for the course and save this as `app/server.mjs`. It needs only Node.js, no packages:

```javascript
// A tiny app with four kinds of flakiness built in. Run: node app/server.mjs
import http from 'node:http';

const PORT = Number(process.env.PORT || 4817);
const sleep = (ms) => new Promise((r) => setTimeout(r, ms));
const random = (min, max) => min + Math.random() * (max - min);

let users = []; // shared by every test that talks to this server
const themes = {}; // theme per account; tests that share an account share it

const page = (body, theme) => `<!doctype html><html><head><meta charset="utf-8"><title>Flaky demo</title>
<style>.dark{background:#222;color:#eee}</style></head><body class="${theme}">${body}</body></html>`;

const pages = {
  // 1. Timing: the save takes 100 to 800 ms.
  '/save': `<button id="save">Save</button><p id="status" role="status"></p>
<script>
document.getElementById('save').onclick = async () => {
  await fetch('/api/save', { method: 'POST' });
  document.getElementById('status').textContent = 'Saved';
};
</script>`,
  // 2. Shared data: one user list on the server.
  '/users': `<form id="f"><label>Name <input name="name"></label><button>Add user</button></form>
<ul id="list"></ul>
<script>
const render = async () => {
  const users = await (await fetch('/api/users')).json();
  // textContent, not innerHTML: names come from users, so they must never be parsed as HTML
  document.getElementById('list').replaceChildren(...users.map((u) => {
    const li = document.createElement('li');
    li.textContent = u;
    return li;
  }));
};
document.getElementById('f').onsubmit = async (e) => {
  e.preventDefault();
  await fetch('/api/users', { method: 'POST', body: e.target.name.value });
  e.target.reset();
  render();
};
render();
</script>`,
  // 3. Order: a theme setting stored on the server, per account (?account=...).
  '/settings': `<label><input type="checkbox" id="dark"> Dark mode</label>
<script>
const box = document.getElementById('dark');
box.checked = document.body.className === 'dark';
box.onchange = async () => {
  await fetch('/api/theme' + location.search, { method: 'POST', body: box.checked ? 'dark' : 'light' });
  document.body.className = box.checked ? 'dark' : 'light';
};
</script>`,
  // 4. A real bug: search results can arrive out of order.
  '/search': `<label>Search <input id="q"></label><ul id="results" aria-busy="false"></ul>
<script>
const list = document.getElementById('results');
let pending = 0;
document.getElementById('q').oninput = async (e) => {
  list.ariaBusy = String(++pending > 0);
  const res = await fetch('/api/search?q=' + encodeURIComponent(e.target.value));
  const items = await res.json();
  list.replaceChildren(...items.map((i) => {
    const li = document.createElement('li');
    li.textContent = i;
    return li;
  }));
  list.ariaBusy = String(--pending > 0);
};
</script>`,
};

const LIBRARIES = ['playwright', 'puppeteer', 'pytest', 'selenium', 'cypress', 'appium'];

const server = http.createServer(async (req, res) => {
  const url = new URL(req.url, `http://${req.headers.host}`);
  const body = await new Promise((r) => { let b = ''; req.on('data', (c) => (b += c)); req.on('end', () => r(b)); });
  const json = (data) => { res.setHeader('content-type', 'application/json'); res.end(JSON.stringify(data)); };

  const account = url.searchParams.get('account') || 'demo';
  if (pages[url.pathname]) {
    res.setHeader('content-type', 'text/html');
    return res.end(page(pages[url.pathname], themes[account] || 'light'));
  }
  if (url.pathname === '/api/save') { await sleep(random(100, 800)); return json({ ok: true }); }
  if (url.pathname === '/api/users' && req.method === 'POST') { users.push(body); return json({ ok: true }); }
  if (url.pathname === '/api/users' && req.method === 'DELETE') { users = []; return json({ ok: true }); }
  if (url.pathname === '/api/users') return json(users);
  if (url.pathname === '/api/theme') { themes[account] = body; return json({ ok: true }); }
  if (url.pathname === '/api/search') {
    const q = url.searchParams.get('q') || '';
    await sleep(random(0, 1000));
    return json(LIBRARIES.filter((l) => l.startsWith(q)));
  }
  res.statusCode = 404; res.end('not found');
});

server.listen(PORT, () => console.log(`Flaky demo on http://localhost:${PORT}`));
```

Chapter 6 adds a fifth page, `/search-fixed`, with the race condition repaired.

The server keeps its state in memory for as long as it runs. That is deliberate: a real test environment also remembers what earlier tests did, and several examples depend on it. When a measurement needs a clean start, stop the server and start it again.

## The Playwright project[#](https://testingbot.com#the-playwright-project)

```bash
npm init -y
npm i -D @playwright/test
npx playwright install chromium
```

`playwright.config.ts` starts the demo app for you and runs tests in parallel:

```typescript
import { defineConfig } from '@playwright/test';

export default defineConfig({
  testDir: './tests',
  fullyParallel: true,
  retries: 0,
  reporter: [['list'], ['json', { outputFile: 'results.json' }]],
  use: { baseURL: 'http://localhost:4817', trace: 'retain-on-failure' },
  webServer: {
    command: 'node app/server.mjs',
    url: 'http://localhost:4817/save',
    reuseExistingServer: !process.env.CI,
  },
});
```

`retries: 0` is on purpose for now. While you measure, every failure must be visible; chapter 7 turns retries back on.

## The Python project[#](https://testingbot.com#the-python-project)

```bash
python3 -m venv .venv
.venv/bin/pip install selenium pytest pytest-repeat pytest-randomly pytest-rerunfailures pytest-xdist
```

The versions used for this course were Selenium 4.52.0, pytest 9.2.0, pytest-repeat 0.9.4, pytest-randomly 5.1.0, pytest-rerunfailures 16.7 and pytest-xdist 3.8.0. Put the Python tests in their own folder (`py/` here) with this `conftest.py`. It starts headless Chrome for each test; chapter 8 extends it with a `--grid` option:

```python
import os

import pytest
from selenium import webdriver

BASE = os.environ.get("BASE_URL", "http://localhost:4817")

@pytest.fixture
def base():
    return BASE

@pytest.fixture
def driver():
    options = webdriver.ChromeOptions()
    options.add_argument("--headless=new")
    d = webdriver.Chrome(options=options)
    yield d
    d.quit()
```

pytest does not start the app, so run `node app/server.mjs` in a second terminal before the Python tests.

## Run one test many times[#](https://testingbot.com#run-one-test-many-times)

The basic tool is repetition. Both frameworks can run a test many times in one command:

```bash
# Playwright: every matching test 50 times, 4 at a time
npx playwright test tests/timing.spec.ts --repeat-each 50 --workers 4

# pytest-repeat: every test 30 times; pytest-xdist: 4 processes
.venv/bin/pytest test_timing.py --count 30 -n 4
```

Playwright's report lists each repetition as its own test. To count passes per test, read the JSON report. This small script prints passes over runs for each test title:

```javascript
// rates.mjs: node rates.mjs results.json
import fs from 'node:fs';
const r = JSON.parse(fs.readFileSync(process.argv[2] || 'results.json'));
const stats = {};
const walk = (s) => {
  for (const spec of s.specs || []) for (const t of spec.tests) for (const res of t.results) {
    stats[spec.title] ??= { pass: 0, total: 0 };
    stats[spec.title].total++;
    if (res.status === 'passed') stats[spec.title].pass++;
  }
  for (const c of s.suites || []) walk(c);
};
r.suites.forEach(walk);
for (const [k, v] of Object.entries(stats)) console.log(`${v.pass}/${v.total} ${k}`);
```

```bash
npx playwright test tests/timing.spec.ts --repeat-each 50 --workers 4 --reporter json > results.json
node rates.mjs results.json
```

For pytest, `-rA` prints one line per test with its outcome, which is easy to count with `grep` and `uniq -c`.

Repeat both ways: alone (`--workers 1`, no `-n`) and in parallel. Chapter 4 shows a test that passed 50 of 50 runs with one worker and 0 of 50 with four. The parallel number is the one CI will see.

## How many runs are enough[#](https://testingbot.com#how-many-runs-are-enough)

Suppose a test fails 5% of the time. The chance that it passes 10 runs in a row anyway is 0.95 to the power 10, about 60%. Ten green runs after a "fix" therefore tell you very little. Two rules of thumb:

- To be about 95% sure a failure rate of _p_ is gone, you need roughly 3 / _p_ consecutive passes (the "rule of three"). For a 5% flake that is about 60 runs; for a 1% flake, about 300.
- Measure the failure rate before the fix with the same command you will use after it. A test that failed 25 of 50 times before and 0 of 50 after is convincing. A test that failed once in the last month and then passed 20 times proves nothing yet.

Cheap tests can run hundreds of times locally. Slow tests may need an overnight loop or a scheduled CI job that runs one suspect test many times.

A test failed in roughly 1 of 50 CI runs last month. You changed a wait and it then passed 20 runs in a row. What can you conclude?

 The fix worked: 20 green runs is far more than the 1 failure Very little: a 2% flake passes 20 runs in a row about two times in three anyway The fix made it worse, because the test is now slower

0.98 to the power 20 is about 0.67, so a test with an unchanged 2% failure rate would usually pass 20 runs in a row. To show a 2% flake is gone you need on the order of 150 passes (3 / 0.02).

## Expose order and parallel problems on purpose[#](https://testingbot.com#expose-order-and-parallel-problems-on-purpose)

Repeating one test finds timing problems. Some flakiness only appears when tests run in a different order or at the same time, so measure that too:

```bash
# Playwright 1.64+: random order; the seed is printed so you can replay it
npx playwright test --shuffle
npx playwright test --shuffle 540272254

# pytest-randomly shuffles by default once installed; replay with the printed seed
.venv/bin/pytest -p randomly --randomly-seed=3183623594
.venv/bin/pytest -p no:randomly # file order, for comparison
```

`--repeat-each` also changes the order indirectly: the second copy of a test runs after the first copy of every other test. Chapter 5 uses exactly that to expose a test that depends on order.

## What retries tell you[#](https://testingbot.com#what-retries-tell-you)

When retries are on, Playwright marks a test that failed and then passed on a retry as **flaky** in its report, separately from passed and failed. pytest-rerunfailures prints a count of reruns in the summary line (for example `4 failed, 16 passed, 18 rerun`). Both are measurements you get for free on every CI run. Chapter 7 covers how to use them without letting them hide failures.

## History beats a single run[#](https://testingbot.com#history-beats-a-single-run)

The local measurements answer "how flaky is this test now". CI history answers "since when, on which browser, with which error". On TestingBot, each test's detail page has a History strip with its last 20 runs, and the [All runs](https://testingbot.com/support/analytics/insights) page labels a test **Flaky** when, on one browser or device with at least 5 decided runs, between 10% and 90% of the runs failed and the result changed at least twice. The exact rules are on [Metrics and definitions](https://testingbot.com/support/analytics/metrics). History only works when every run of a test uses the same name, which chapter 8 sets up.

Your CI puts the build number in every test name, such as `nightly 1042 - Login`. Why can no history tool call the login test flaky?

 Nightly builds are excluded from history The test needs retries enabled first Every run has a different name, so each run is a separate test with one result History only counts failed runs

History groups runs by exact name. Keep the test name stable and put the build number in the build field instead, so all runs of the login test line up and their flips can be counted.

## A measurement log[#](https://testingbot.com#a-measurement-log)

Keep a short note per flaky test: the command you ran, the pass rate, the error, and the environment. For the save test in the next chapter it reads:

| Test | Command | Before | After |
| --- | --- | --- | --- |
| save shows a confirmation | `--repeat-each 50 --workers 4` | 25/50 | 50/50 |

That table goes into the pull request that fixes the test. It is the difference between "should be fixed" and "is fixed".

[Previous · Chapter 1 Flaky Tests: How to Find, Fix and Prevent Them](https://testingbot.com/resources/courses/flaky-tests) [Next · Chapter 3 Root Cause 1: Timing, Sleeps and Waits](https://testingbot.com/resources/courses/flaky-tests/timing-and-waits)

On this page

1. [The demo app](https://testingbot.com#the-demo-app)
2. [The Playwright project](https://testingbot.com#the-playwright-project)
3. [The Python project](https://testingbot.com#the-python-project)
4. [Run one test many times](https://testingbot.com#run-one-test-many-times)
5. [How many runs are enough](https://testingbot.com#how-many-runs-are-enough)
6. [Expose order and parallel problems on purpose](https://testingbot.com#expose-order-and-parallel-problems-on-purpose)
7. [What retries tell you](https://testingbot.com#what-retries-tell-you)
8. [History beats a single run](https://testingbot.com#history-beats-a-single-run)
9. [A measurement log](https://testingbot.com#a-measurement-log)

Certification

Earn the free, resume-worthy TestingBot Flaky Test Fixing Certification for this course.

[How it works](https://testingbot.com/resources/courses/flaky-tests/certification)
