Exit course
Chapter 3 of 10 Intermediate 8 min read 2 quick checks Updated

Root Cause 1: Timing, Sleeps and Waits

On this page
  1. A reproducible flaky test
  2. The fix: wait for the state, not for time
  3. The same test in Selenium
  4. A Selenium-only flake: stale elements
  5. Waiting for the network
  6. When a timeout is genuinely too short
  7. Checklist

The most common flaky test checks the page before the page is ready. The test and the browser run concurrently: your test sends a click, the app starts a request, and the test moves on to its next line while the request is still in flight. Whether the next check passes depends on which side wins the race.

A reproducible flaky test#

The demo app's /save page has a button that calls an API, and the API takes between 100 and 800 ms. When the response comes back, the page shows "Saved". Here is the test most people write first, in tests/timing.spec.ts:

import { test, expect } from '@playwright/test';

test('save shows a confirmation (flaky)', async ({ page }) => {
  await page.goto('/save');
  await page.getByRole('button', { name: 'Save' }).click();
  await page.waitForTimeout(500);
  expect(await page.getByRole('status').textContent()).toBe('Saved');
});

It passes on the developer's machine most of the time, which is how it gets merged. Run it 50 times:

npx playwright test tests/timing.spec.ts -g flaky --repeat-each 50 --workers 4 --reporter json > results.json
node rates.mjs results.json

In the run for this course it passed 25 of 50. The failures all look the same:

Expected: "Saved"
Received: ""

The test has two separate problems, and both matter:

  1. A fixed sleep. 500 ms is a guess about how long the save takes. When the API takes 600 ms, the guess is wrong. On a busy CI runner or a remote browser, more runs land on the wrong side of the guess.
  2. A one-shot read. textContent() reads the element once and returns whatever is there. expect(...).toBe() on that plain string cannot retry. Even with a longer sleep, the check is a snapshot of one moment.

The fix: wait for the state, not for time#

test('save shows a confirmation (fixed)', async ({ page }) => {
  await page.goto('/save');
  await page.getByRole('button', { name: 'Save' }).click();
  await expect(page.getByRole('status')).toHaveText('Saved');
});

toHaveText on a locator is a web-first assertion: it re-reads the element until the text matches or the assertion timeout (5 seconds by default) runs out. When the save takes 120 ms, the test continues after about 120 ms; when it takes 780 ms, after about 780 ms. Same command, 50 runs: 50 of 50 passed.

The fix also made the test faster on average, because it no longer sleeps for 500 ms when the save took 150. Fixed sleeps are always either too short (flaky) or too long (slow), usually both, depending on the run.

The Playwright course's chapter on auto-waiting and web-first assertions lists the full set of retrying matchers. The rule to remember: if the value comes from the page, assert on the locator, not on a value you read from it.

A teammate fixes the flaky save test by changing waitForTimeout(500) to waitForTimeout(2000). It now passes 50 of 50 runs. Why is that still the wrong fix?

A longer sleep trades flakiness for slowness and only moves the threshold. A retrying assertion waits exactly as long as needed and fails with a clear message when the state never arrives.

The same test in Selenium#

Selenium commands do not wait for the page to change. The flaky version looks the same as in Playwright, in py/test_timing.py:

import time
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.support.ui import WebDriverWait


def test_save_flaky(driver, base):
    driver.get(f"{base}/save")
    driver.find_element(By.ID, "save").click()
    time.sleep(0.5)
    assert driver.find_element(By.ID, "status").text == "Saved"


def test_save_fixed(driver, base):
    driver.get(f"{base}/save")
    driver.find_element(By.ID, "save").click()
    WebDriverWait(driver, 5).until(
        EC.text_to_be_present_in_element((By.ID, "status"), "Saved"))
.venv/bin/pytest -p no:randomly test_timing.py --count 30 -n 4 -rA

Result: the sleep version passed 18 of 30, the explicit wait 30 of 30. WebDriverWait.until polls the condition (every 500 ms by default) until it returns something truthy or the timeout expires, which makes it Selenium's equivalent of a retrying assertion. Note that EC.text_to_be_present_in_element checks that the text contains "Saved", not that it equals it; write a small lambda when you need equality. The Selenium Python course covers the expected conditions in detail.

Two Selenium habits make this worse:

  • Implicit waits mixed with explicit waits. driver.implicitly_wait(10) makes every find_element wait for presence. Combined with WebDriverWait, the timeouts interact and a failing wait can take far longer than either value. The Selenium documentation advises against mixing them; pick explicit waits.
  • Waiting for presence when you need something else. An element can be present in the DOM but hidden, disabled or still empty. Wait for the condition the user would wait for: visible, clickable, or containing the expected text.

A Selenium-only flake: stale elements#

Playwright locators are re-resolved every time they are used. Selenium WebElement objects are handles to one specific DOM node, and when the page replaces that node, the handle goes stale. Any page that re-renders a list is a candidate.

The demo app's /users page replaces every <li> in the list with new elements each time it reloads it. This wait looks correct:

def list_items(driver):
    return [li.text for li in driver.find_elements(By.CSS_SELECTOR, "#list li")]

WebDriverWait(driver, 5).until(lambda d: name in list_items(d))

In two batches of 20 runs, one with a single process and one with -n 4, it passed 10 times each. The failures were not timeouts but StaleElementReferenceException: find_elements returned the old <li> nodes, the page re-rendered, and reading .text from an old node raised. WebDriverWait only ignores NoSuchElementException by default, so the stale error ends the wait immediately.

Two fixes, both measured at 20 of 20:

from selenium.common.exceptions import StaleElementReferenceException

# 1. Read the text in one call, from an element that is not replaced
def list_items(driver):
    return driver.find_element(By.ID, "list").text.splitlines()

# 2. Or tell the wait to retry on stale elements too
wait = WebDriverWait(driver, 5, ignored_exceptions=[StaleElementReferenceException])
wait.until(lambda d: name in [li.text for li in d.find_elements(By.CSS_SELECTOR, "#list li")])

The first is simpler and also faster. The general rule: never keep a WebElement across a change you triggered; find it again after the change.

A Selenium wait WebDriverWait(driver, 10).until(lambda d: d.find_element(By.ID, "total").text == "3") sometimes fails within a second with StaleElementReferenceException. What is happening?

A re-render can happen between finding the element and reading it. WebDriverWait only swallows NoSuchElementException unless you pass ignored_exceptions; the stale error ends the wait at once, long before the timeout.

Waiting for the network#

Sometimes the signal you need is not on the page, for example when a test must check the payload the app sent. Wait for that specific response, and start waiting before you trigger it:

const saved = page.waitForResponse((r) => r.url().endsWith('/api/save'));
await page.getByRole('button', { name: 'Save' }).click();
expect((await saved).ok()).toBe(true);

Starting the wait after the click is itself a race: a fast response can arrive before the wait begins, and the test then waits for an event that already happened.

Avoid waitForLoadState('networkidle') as a general "page is ready" signal. Playwright's documentation discourages it for testing, because apps that poll or keep connections open may never go idle, and an idle network does not mean the page finished rendering.

When a timeout is genuinely too short#

Retrying assertions do not remove timeouts; they make them an upper bound. If an operation really takes 8 seconds, a 5 second assertion timeout fails every time, which is a deterministic failure, not a flaky one. A test that fails only on slow runs, near the limit, is different: look at the durations of its passing runs first. If passes take 4.5 seconds against a 5 second limit, raise the timeout for that one assertion, { timeout: 15_000 }, and leave the global default alone so the rest of the suite still fails fast. If the passing runs take 200 ms and the failures hit the limit, the problem is not the timeout: something is never arriving, and the next chapters cover why.

Checklist#

  • No fixed sleeps (waitForTimeout, time.sleep, Thread.sleep) in tests. Search for them in code review.
  • In Playwright, assert on locators with await expect(locator)..., not on values read from them.
  • In Selenium, use WebDriverWait with the condition the user would wait for, and do not mix it with implicit waits.
  • Re-find Selenium elements after anything that can re-render them.
  • Start waiting for an event before you trigger it.