Root Cause 3: Test Order and Isolation
On this page
Some tests pass only because of what ran before them, or fail only because of it. The suite looks stable for months because the order never changes, and then someone adds a test, renames a file, raises the worker count or turns on sharding, and an unrelated test starts failing. The test that fails is the victim; the cause is the test that left something behind, sometimes called the polluter.
A reproducible example#
The demo app's /settings page has a dark mode switch, stored on the server per account. Two reasonable tests, in tests/settings.spec.ts:
import { test, expect } from '@playwright/test';
test('new accounts start with the light theme (flaky)', async ({ page }) => {
await page.goto('/settings');
await expect(page.locator('body')).toHaveClass('light');
});
test('dark mode can be switched on (flaky)', async ({ page }) => {
await page.goto('/settings');
await page.getByLabel('Dark mode').check();
await expect(page.locator('body')).toHaveClass('dark');
});
Both use the default demo account, like a suite that logs in as one shared test user. With a fresh server and one worker, the file runs top to bottom and both pass. Change only the order or the count, and the first test fails:
| Command | "light theme" test |
|---|---|
--workers 1 |
1 of 1 |
--workers 1 --repeat-each 2 |
1 of 2 |
--workers 4 --repeat-each 25 |
2 of 25 |
--workers 1 --shuffle, 20 separate runs |
10 of 20 runs failed |
With --repeat-each 2, the second copy of the light theme test runs after the dark mode test switched the shared account to dark. With --shuffle, it depends on the seed. Nothing about either test is wrong on its own; together, they depend on order.
Playwright prints the seed of a shuffled run, so a failing order can be replayed exactly:
npx playwright test tests/settings.spec.ts --workers 1 --shuffle
# Running 2 tests using 1 worker, shuffle seed 540272254
npx playwright test tests/settings.spec.ts --workers 1 --shuffle 540272254
With seed 540272254 the dark mode test runs first, and the light theme test fails every time you replay it. --shuffle was added in Playwright 1.64. It shuffles test files, and the tests inside a file when they run in parallel mode (fullyParallel: true or test.describe.configure({ mode: 'parallel' })); tests in a serial group keep their order.
A shuffled Playwright run fails, and the next run passes. What do you do first?
The seed turns an intermittent order problem into a reproducible one. Shuffling did not cause the failure; it revealed a dependency between tests that a fixed order was hiding.
The same thing in pytest#
pytest-randomly shuffles the test order on every run once it is installed, and prints the seed. The Selenium version, in py/test_settings.py, already includes the fixed pair that the next section explains:
import uuid
import pytest
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
def test_new_account_is_light_flaky(driver, base):
driver.get(f"{base}/settings")
assert driver.find_element(By.TAG_NAME, "body").get_attribute("class") == "light"
def test_dark_mode_flaky(driver, base):
driver.get(f"{base}/settings")
driver.find_element(By.ID, "dark").click()
WebDriverWait(driver, 5).until(
lambda d: d.find_element(By.TAG_NAME, "body").get_attribute("class") == "dark")
@pytest.fixture
def account():
return f"test-{uuid.uuid4()}"
def test_new_account_is_light_fixed(driver, base, account):
driver.get(f"{base}/settings?account={account}")
assert driver.find_element(By.TAG_NAME, "body").get_attribute("class") == "light"
def test_dark_mode_fixed(driver, base, account):
driver.get(f"{base}/settings?account={account}")
driver.find_element(By.ID, "dark").click()
WebDriverWait(driver, 5).until(
lambda d: d.find_element(By.TAG_NAME, "body").get_attribute("class") == "dark")
For the flaky pair (-k flaky): in file order (-p no:randomly) on a fresh server, both pass. In 20 shuffled runs, each on a freshly started server, 11 failed. One failing seed:
.venv/bin/pytest test_settings.py -k flaky --randomly-seed=3183623594
Replay that seed twice without restarting the server and the result changes again: the first replay fails the light theme test, and the second fails the dark mode test. The server still remembered dark mode from the previous run, so the checkbox started checked and the click switched it off. State leaks between runs, not only between tests, which is why the seed only reproduces an order failure when the environment is also reset.
Why Playwright and Selenium leak different things#
Some isolation comes from the framework, and the two differ:
-
Browser state. Playwright Test creates a new browser context for every test: fresh cookies, local storage, session storage and cache. A test cannot inherit a login or a cart from the test before it. In Selenium, isolation depends on your fixture. A function-scoped
driverfixture (as in this course) starts a clean browser per test. A module- or session-scoped driver, popular because starting browsers is slow, carries cookies, local storage and the current page from one test to the next. If you share a driver for speed, clear its state between tests withdriver.delete_all_cookies()and a script that clearslocalStorageandsessionStoragefor the app's origin. - Server state. No framework isolates this. The settings example fails in Playwright despite fresh contexts, because the theme lives on the server. Fresh browser contexts remove one category of order dependence, not all of them.
- Process state. Module-level variables in a test file are shared by every test that runs in the same worker process, in both pytest and Playwright. A cache or counter at the top of a test file is shared state too.
The fix: each test sets up the state it needs#
The robust fix is the same as in the previous chapter: give each test its own account, so it starts from a known state no matter what ran before:
test.describe('fixed', () => {
let account: string;
test.beforeEach(async ({}, testInfo) => {
account = `test-${testInfo.testId}-${testInfo.repeatEachIndex}-${testInfo.retry}`;
});
test('new accounts start with the light theme (fixed)', async ({ page }) => {
await page.goto(`/settings?account=${account}`);
await expect(page.locator('body')).toHaveClass('light');
});
test('dark mode can be switched on (fixed)', async ({ page }) => {
await page.goto(`/settings?account=${account}`);
await page.getByLabel('Dark mode').check();
await expect(page.locator('body')).toHaveClass('dark');
});
});
With --workers 4 --repeat-each 25 --shuffle: 25 of 25 for both. In pytest, the account fixture above does the same; the fixed pair (-k fixed) failed in 0 of 20 shuffled runs.
When a test cannot have its own copy (one global feature flag, one shared device), make the dependency explicit instead of implicit:
- The test sets the state it needs in its own setup, rather than assuming the default.
- Tests that change global state run in a serial group or a separate project with one worker, so nothing runs alongside them.
- The test restores what it changed in its teardown. This alone is not enough under parallelism, but it stops one failed test from poisoning every later run.
To fix the theme tests, a teammate puts them in one file with test.describe.configure({ mode: 'serial' }) so that the light theme test always runs first. What is the weakness of that fix?
Ordering the two tests protects them from each other only. The light theme test still assumes the shared account is light; an own account per test removes the assumption.
Finding the polluter#
When a test fails only in the full suite, you need the test that runs before it and leaves state behind. Three techniques:
-
Bisect the order. Replay the failing seed with half of the earlier tests removed. If it still fails, the polluter is in the remaining half. Playwright's
--test-listoption takes a file listing the tests to run, which makes this easy to script; with pytest, pass the test IDs on the command line. - Diff the state. Dump the relevant state (the account's settings, the rows the test reads) at the start of the failing test in a passing and a failing order, and compare.
-
Run the victim right after each suspect. For a handful of candidates, run
suspect, victimpairs in serial mode.
Then fix the victim so it no longer depends on the state, and fix the polluter so it no longer leaves it behind. Fixing only one of them leaves the next pair waiting to happen.
Checklist#
- Run shuffled (
--shuffle, pytest-randomly) regularly, for example in a nightly job, and keep the seed of every failure. - Every test sets up its own preconditions; none relies on defaults another test might have changed.
- Know where your isolation comes from: fresh browser contexts or a function-scoped driver for browser state, unique accounts for server state.
- Restart or reset the test environment between runs, so a replayed seed reproduces the same result.