Exit course
Chapter 2 of 10 Intermediate 10 min read 2 quick checks Updated

Measure Flakiness Before You Fix It

On this page
  1. The demo app
  2. The Playwright project
  3. The Python project
  4. Run one test many times
  5. How many runs are enough
  6. Expose order and parallel problems on purpose
  7. What retries tell you
  8. History beats a single run
  9. A measurement log

A flaky test fails some of the time. Before you change anything, find out how often, under which conditions, and with which error. That number is what tells you later whether your fix worked. This chapter sets up the demo app and the two test projects, then shows the measuring tools used in every later chapter.

The demo app#

Create a folder for the course and save this as app/server.mjs. It needs only Node.js, no packages:

// A tiny app with four kinds of flakiness built in. Run: node app/server.mjs
import http from 'node:http';

const PORT = Number(process.env.PORT || 4817);
const sleep = (ms) => new Promise((r) => setTimeout(r, ms));
const random = (min, max) => min + Math.random() * (max - min);

let users = [];                  // shared by every test that talks to this server
const themes = {};               // theme per account; tests that share an account share it

const page = (body, theme) => `<!doctype html><html><head><meta charset="utf-8"><title>Flaky demo</title>
<style>.dark{background:#222;color:#eee}</style></head><body class="${theme}">${body}</body></html>`;

const pages = {
  // 1. Timing: the save takes 100 to 800 ms.
  '/save': `<button id="save">Save</button><p id="status" role="status"></p>
<script>
document.getElementById('save').onclick = async () => {
  await fetch('/api/save', { method: 'POST' });
  document.getElementById('status').textContent = 'Saved';
};
</script>`,
  // 2. Shared data: one user list on the server.
  '/users': `<form id="f"><label>Name <input name="name"></label><button>Add user</button></form>
<ul id="list"></ul>
<script>
const render = async () => {
  const users = await (await fetch('/api/users')).json();
  // textContent, not innerHTML: names come from users, so they must never be parsed as HTML
  document.getElementById('list').replaceChildren(...users.map((u) => {
    const li = document.createElement('li');
    li.textContent = u;
    return li;
  }));
};
document.getElementById('f').onsubmit = async (e) => {
  e.preventDefault();
  await fetch('/api/users', { method: 'POST', body: e.target.name.value });
  e.target.reset();
  render();
};
render();
</script>`,
  // 3. Order: a theme setting stored on the server, per account (?account=...).
  '/settings': `<label><input type="checkbox" id="dark"> Dark mode</label>
<script>
const box = document.getElementById('dark');
box.checked = document.body.className === 'dark';
box.onchange = async () => {
  await fetch('/api/theme' + location.search, { method: 'POST', body: box.checked ? 'dark' : 'light' });
  document.body.className = box.checked ? 'dark' : 'light';
};
</script>`,
  // 4. A real bug: search results can arrive out of order.
  '/search': `<label>Search <input id="q"></label><ul id="results" aria-busy="false"></ul>
<script>
const list = document.getElementById('results');
let pending = 0;
document.getElementById('q').oninput = async (e) => {
  list.ariaBusy = String(++pending > 0);
  const res = await fetch('/api/search?q=' + encodeURIComponent(e.target.value));
  const items = await res.json();
  list.replaceChildren(...items.map((i) => {
    const li = document.createElement('li');
    li.textContent = i;
    return li;
  }));
  list.ariaBusy = String(--pending > 0);
};
</script>`,
};

const LIBRARIES = ['playwright', 'puppeteer', 'pytest', 'selenium', 'cypress', 'appium'];

const server = http.createServer(async (req, res) => {
  const url = new URL(req.url, `http://${req.headers.host}`);
  const body = await new Promise((r) => { let b = ''; req.on('data', (c) => (b += c)); req.on('end', () => r(b)); });
  const json = (data) => { res.setHeader('content-type', 'application/json'); res.end(JSON.stringify(data)); };

  const account = url.searchParams.get('account') || 'demo';
  if (pages[url.pathname]) {
    res.setHeader('content-type', 'text/html');
    return res.end(page(pages[url.pathname], themes[account] || 'light'));
  }
  if (url.pathname === '/api/save') { await sleep(random(100, 800)); return json({ ok: true }); }
  if (url.pathname === '/api/users' && req.method === 'POST') { users.push(body); return json({ ok: true }); }
  if (url.pathname === '/api/users' && req.method === 'DELETE') { users = []; return json({ ok: true }); }
  if (url.pathname === '/api/users') return json(users);
  if (url.pathname === '/api/theme') { themes[account] = body; return json({ ok: true }); }
  if (url.pathname === '/api/search') {
    const q = url.searchParams.get('q') || '';
    await sleep(random(0, 1000));
    return json(LIBRARIES.filter((l) => l.startsWith(q)));
  }
  res.statusCode = 404; res.end('not found');
});

server.listen(PORT, () => console.log(`Flaky demo on http://localhost:${PORT}`));

Chapter 6 adds a fifth page, /search-fixed, with the race condition repaired.

The server keeps its state in memory for as long as it runs. That is deliberate: a real test environment also remembers what earlier tests did, and several examples depend on it. When a measurement needs a clean start, stop the server and start it again.

The Playwright project#

npm init -y
npm i -D @playwright/test
npx playwright install chromium

playwright.config.ts starts the demo app for you and runs tests in parallel:

import { defineConfig } from '@playwright/test';

export default defineConfig({
  testDir: './tests',
  fullyParallel: true,
  retries: 0,
  reporter: [['list'], ['json', { outputFile: 'results.json' }]],
  use: { baseURL: 'http://localhost:4817', trace: 'retain-on-failure' },
  webServer: {
    command: 'node app/server.mjs',
    url: 'http://localhost:4817/save',
    reuseExistingServer: !process.env.CI,
  },
});

retries: 0 is on purpose for now. While you measure, every failure must be visible; chapter 7 turns retries back on.

The Python project#

python3 -m venv .venv
.venv/bin/pip install selenium pytest pytest-repeat pytest-randomly pytest-rerunfailures pytest-xdist

The versions used for this course were Selenium 4.52.0, pytest 9.2.0, pytest-repeat 0.9.4, pytest-randomly 5.1.0, pytest-rerunfailures 16.7 and pytest-xdist 3.8.0. Put the Python tests in their own folder (py/ here) with this conftest.py. It starts headless Chrome for each test; chapter 8 extends it with a --grid option:

import os

import pytest
from selenium import webdriver

BASE = os.environ.get("BASE_URL", "http://localhost:4817")


@pytest.fixture
def base():
    return BASE


@pytest.fixture
def driver():
    options = webdriver.ChromeOptions()
    options.add_argument("--headless=new")
    d = webdriver.Chrome(options=options)
    yield d
    d.quit()

pytest does not start the app, so run node app/server.mjs in a second terminal before the Python tests.

Run one test many times#

The basic tool is repetition. Both frameworks can run a test many times in one command:

# Playwright: every matching test 50 times, 4 at a time
npx playwright test tests/timing.spec.ts --repeat-each 50 --workers 4

# pytest-repeat: every test 30 times; pytest-xdist: 4 processes
.venv/bin/pytest test_timing.py --count 30 -n 4

Playwright's report lists each repetition as its own test. To count passes per test, read the JSON report. This small script prints passes over runs for each test title:

// rates.mjs: node rates.mjs results.json
import fs from 'node:fs';
const r = JSON.parse(fs.readFileSync(process.argv[2] || 'results.json'));
const stats = {};
const walk = (s) => {
  for (const spec of s.specs || []) for (const t of spec.tests) for (const res of t.results) {
    stats[spec.title] ??= { pass: 0, total: 0 };
    stats[spec.title].total++;
    if (res.status === 'passed') stats[spec.title].pass++;
  }
  for (const c of s.suites || []) walk(c);
};
r.suites.forEach(walk);
for (const [k, v] of Object.entries(stats)) console.log(`${v.pass}/${v.total}  ${k}`);
npx playwright test tests/timing.spec.ts --repeat-each 50 --workers 4 --reporter json > results.json
node rates.mjs results.json

For pytest, -rA prints one line per test with its outcome, which is easy to count with grep and uniq -c.

Repeat both ways: alone (--workers 1, no -n) and in parallel. Chapter 4 shows a test that passed 50 of 50 runs with one worker and 0 of 50 with four. The parallel number is the one CI will see.

How many runs are enough#

Suppose a test fails 5% of the time. The chance that it passes 10 runs in a row anyway is 0.95 to the power 10, about 60%. Ten green runs after a "fix" therefore tell you very little. Two rules of thumb:

  • To be about 95% sure a failure rate of p is gone, you need roughly 3 / p consecutive passes (the "rule of three"). For a 5% flake that is about 60 runs; for a 1% flake, about 300.
  • Measure the failure rate before the fix with the same command you will use after it. A test that failed 25 of 50 times before and 0 of 50 after is convincing. A test that failed once in the last month and then passed 20 times proves nothing yet.

Cheap tests can run hundreds of times locally. Slow tests may need an overnight loop or a scheduled CI job that runs one suspect test many times.

A test failed in roughly 1 of 50 CI runs last month. You changed a wait and it then passed 20 runs in a row. What can you conclude?

0.98 to the power 20 is about 0.67, so a test with an unchanged 2% failure rate would usually pass 20 runs in a row. To show a 2% flake is gone you need on the order of 150 passes (3 / 0.02).

Expose order and parallel problems on purpose#

Repeating one test finds timing problems. Some flakiness only appears when tests run in a different order or at the same time, so measure that too:

# Playwright 1.64+: random order; the seed is printed so you can replay it
npx playwright test --shuffle
npx playwright test --shuffle 540272254

# pytest-randomly shuffles by default once installed; replay with the printed seed
.venv/bin/pytest -p randomly --randomly-seed=3183623594
.venv/bin/pytest -p no:randomly      # file order, for comparison

--repeat-each also changes the order indirectly: the second copy of a test runs after the first copy of every other test. Chapter 5 uses exactly that to expose a test that depends on order.

What retries tell you#

When retries are on, Playwright marks a test that failed and then passed on a retry as flaky in its report, separately from passed and failed. pytest-rerunfailures prints a count of reruns in the summary line (for example 4 failed, 16 passed, 18 rerun). Both are measurements you get for free on every CI run. Chapter 7 covers how to use them without letting them hide failures.

History beats a single run#

The local measurements answer "how flaky is this test now". CI history answers "since when, on which browser, with which error". On TestingBot, each test's detail page has a History strip with its last 20 runs, and the All runs page labels a test Flaky when, on one browser or device with at least 5 decided runs, between 10% and 90% of the runs failed and the result changed at least twice. The exact rules are on Metrics and definitions. History only works when every run of a test uses the same name, which chapter 8 sets up.

Your CI puts the build number in every test name, such as nightly 1042 - Login. Why can no history tool call the login test flaky?

History groups runs by exact name. Keep the test name stable and put the build number in the build field instead, so all runs of the login test line up and their flips can be counted.

A measurement log#

Keep a short note per flaky test: the command you ran, the pass rate, the error, and the environment. For the save test in the next chapter it reads:

Test Command Before After
save shows a confirmation --repeat-each 50 --workers 4 25/50 50/50

That table goes into the pull request that fixes the test. It is the difference between "should be fixed" and "is fixed".