Exit course
Chapter 1 of 10 Intermediate 6 min read Updated

Flaky Tests: How to Find, Fix and Prevent Them

Fix flaky tests for good: measure them, then remove the four root causes (waits, shared data, test order, real app races) in Playwright and Selenium, use retries and quarantine safely, and debug flaky runs on a cloud grid.

What you will learn

  1. Flaky Tests: How to Find, Fix and Prevent Them
  2. Measure Flakiness Before You Fix It
  3. Root Cause 1: Timing, Sleeps and Waits
  4. Root Cause 2: Test Data and Shared State
  5. Root Cause 3: Test Order and Isolation
  6. Root Cause 4: When the App Is the Flaky Part
  7. Retries and Quarantine, Done Right
  8. Debugging Flaky Tests on a Cloud Grid
  9. A Team Process for Flaky Tests
  10. Flaky Tests Knowledge Check
On this page
  1. What a flaky test costs
  2. Where flakiness comes from
  3. Measure first, then fix
  4. How the course is organised
  5. Which framework
  6. Prerequisites

Last reviewed November 2026 against @playwright/test 1.65.0, Selenium 4.52.0 and pytest 9.2.0.

A flaky test is a test that passes and fails on the same code. Nothing in the commit changed, nothing in the test changed, and yet one CI run is red and the re-run is green. That last part is what makes flakiness expensive: the cheapest response to a red build is to press re-run, and every time that works, the team learns that red does not mean broken.

This course treats flakiness as a bug with a cause you can find, not as weather. Every example is a tiny test that really is flaky, measured over many runs before and after the fix, in Playwright (TypeScript) and in Selenium (Python) wherever the two frameworks behave differently.

What a flaky test costs#

The obvious cost is CI time: a pipeline that needs a re-run takes twice as long, and a suite with retries enabled spends minutes re-running tests that should have passed the first time. The larger cost is trust.

  • Real failures get ignored. When a test is known to flake, its failures stop being read. The day it fails for a real reason, someone presses re-run, it passes by luck, and the bug ships.
  • People stop adding tests. If the suite already feels unreliable, writing one more end-to-end test feels like adding noise.
  • Debugging gets harder over time. A flaky test that is retried silently for months collects more causes, and by the time someone looks, nobody remembers when it started.

Large engineering organisations have measured this. In a 2016 post on the Google Testing Blog, Flaky Tests at Google and How We Mitigate Them, Google reported that about 1.5% of all its test runs had a flaky result and that almost 16% of its tests had some level of flakiness. Your numbers will differ, but the shape is common: a small rate per run, spread over a large share of the suite, adds up to many red builds.

Where flakiness comes from#

A failing run has a cause, even when the next run passes. In practice the causes fall into a few groups, and this course takes them one at a time:

Root cause What it looks like Chapter
Timing and waits The test checks the page before the page is ready 3
Test data and shared state Two tests running at the same time change the same record 4
Test order and isolation A test only passes when another test ran before it, or did not 5
The app itself The product has a race condition; the test is right to fail 6

An often-cited study of flaky tests in open source projects, An Empirical Analysis of Flaky Tests (Luo, Hariri, Eloussi and Marinov, FSE 2014), found that asynchronous waits were the most common root cause in its sample of fixed flaky tests, followed by concurrency and test order dependency. The order in this course follows the same idea: check the cheap, common cause first.

The infrastructure can also be the cause: a browser that crashes, a network that drops a request, a CI runner that is overloaded. Those failures are real, but they tend to look different (the session never started, the browser disconnected) and they hit random tests rather than the same one. Chapter 8 shows how to tell them apart on a cloud grid.

Measure first, then fix#

The most common mistake with a flaky test is fixing it by guessing: add a sleep, bump a timeout, run it once, see green, merge. One green run proves almost nothing about a test that fails one time in ten. Chapter 2 sets up the measurements this course uses throughout:

  • running one test many times in a row (--repeat-each in Playwright, --count with pytest-repeat),
  • running the suite in a random order (--shuffle in Playwright, pytest-randomly),
  • reading the "flaky" result that retries produce, and the history of a test across CI runs.

Every fix in this course comes with its numbers. For example, the timing test in chapter 3 passed 25 of 50 runs before the fix and 50 of 50 after.

How the course is organised#

  1. What a flaky test is and what it costs (this page)
  2. Measuring flakiness before you change anything
  3. Root cause 1: timing, sleeps and waits
  4. Root cause 2: test data and shared state
  5. Root cause 3: test order and isolation
  6. Root cause 4: when the app is the flaky part
  7. Retries and quarantine, and when retries hide real bugs
  8. Debugging flaky tests on a cloud grid
  9. A team process: ownership, a flaky budget and CI policy
  10. A knowledge check

The examples run against a small demo app that you start on your own machine. It has four pages, and each one has one kind of flakiness built in on purpose: a save button with a slow response, a shared user list, a theme setting stored on the server, and a search box with a real race condition. Because you control the app, you can reproduce every failure and every fix yourself.

Which framework#

The root causes are the same in every framework; the details differ. Playwright's actions and assertions wait automatically and every test gets a fresh browser context, so some classic Selenium flakiness never happens there, and some new kinds do. Where the code differs, the chapters show both. If you are new to either framework, start with the basics first:

Prerequisites#

  • Node.js 22 or newer for the demo app and the Playwright tests.
  • Python 3.10 or newer and Chrome for the Selenium tests (Selenium Manager downloads the driver).
  • Basic experience with one of the two frameworks.
  • For chapter 8, a TestingBot account; the free trial is enough.

The course is free and needs no account. At the end there is a self-check, and an optional TestingBot certification that asks for a passed exam and the course's fixed suite running green on TestingBot.