Exit course
Chapter 4 of 10 Intermediate 7 min read 2 quick checks Updated

Root Cause 2: Test Data and Shared State

On this page
  1. A reproducible example
  2. The fix: every test owns its data
  3. The same fix in Selenium with pytest-xdist
  4. Accounts, sessions and other shared things
  5. Creating data fast
  6. Checklist

A test that passes alone and fails in the full suite usually shares something with another test: a database row, a user account, a file, a feature flag. Run the tests one at a time and they take turns. Run them in parallel and they change the same thing at the same moment. The failure then depends on how the scheduler happened to interleave them, which is the definition of flaky.

A reproducible example#

The demo app's /users page shows one user list, stored on the server. A natural test resets the list, adds a user, and checks that the list contains exactly that user. In tests/users.spec.ts:

import { test, expect } from '@playwright/test';

test.describe('flaky', () => {
  test.beforeEach(async ({ request }) => {
    await request.delete('/api/users');
  });

  test('adds a user (flaky)', async ({ page }) => {
    await page.goto('/users');
    await page.getByLabel('Name').fill('Ada');
    await page.getByRole('button', { name: 'Add user' }).click();
    await expect(page.getByRole('listitem')).toHaveText(['Ada']);
  });
});

The test uses a web-first assertion, so timing is not the issue. Measure it with different numbers of workers:

npx playwright test tests/users.spec.ts -g flaky --repeat-each 50 --workers 1
npx playwright test tests/users.spec.ts -g flaky --repeat-each 50 --workers 2
npx playwright test tests/users.spec.ts -g flaky --repeat-each 50 --workers 4
Workers Passed
1 50 of 50
2 30 of 50
4 0 of 50

With one worker the test is perfect. With two, it fails 40% of the time with a list that has one Ada too many (another copy of the test added it). With four, there is almost always another copy in the middle of its run, so it never passes. A real suite has many different tests rather than copies of one, so the collisions are rarer and the failures look random. This is the classic "passes locally, fails on CI" test: the developer runs one file, CI runs everything in parallel.

Notice that the beforeEach that deletes all users makes things worse, not better. It cleans up for this test by destroying other tests' data mid-run. A global reset is only safe when nothing else runs at the same time.

A test passes every time when you run its file alone, but fails a few times a week on CI, where the suite runs with eight workers. Which cause should you suspect first?

"Alone it passes, in parallel it sometimes fails" points at shared state. Confirm it by running the file with several workers and repeats locally; if the failure rate rises with the worker count, two tests are touching the same data.

The fix: every test owns its data#

Instead of making the world empty, make the test's own data unique, and assert only on what the test created:

test('adds a user (fixed)', async ({ page }, testInfo) => {
  const name = `Ada ${testInfo.testId}-${testInfo.repeatEachIndex}-${testInfo.retry}`;
  await page.goto('/users');
  await page.getByLabel('Name').fill(name);
  await page.getByRole('button', { name: 'Add user' }).click();
  await expect(page.getByRole('listitem').filter({ hasText: name })).toHaveCount(1);
});

With four workers and 50 repeats: 50 of 50. Two changes made the difference:

  1. Unique data. testInfo.testId is stable per test, repeatEachIndex separates repeats, and retry separates retries. A random UUID works too; the advantage of the deterministic name is that you can find a failed run's data afterwards.
  2. A scoped assertion. The test no longer claims the list has exactly one entry, which it cannot know in a shared environment. It checks that its own user appears exactly once.

One more measurement is worth doing: run the flaky and the fixed test together (--repeat-each 50 --workers 4 on the whole file). The fixed test then passed only 48 of 50: twice, the flaky test's DELETE removed the fixed test's user between its add and its check. A single test with a global reset can make well-written tests flaky, which is why a shared-state problem is often found in a different test than the one that fails.

The same fix in Selenium with pytest-xdist#

pytest-xdist runs tests in several processes with -n. The same collision appears, in py/test_users.py:

import urllib.request
import uuid

from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait


def add_user(driver, base, name):
    driver.get(f"{base}/users")
    driver.find_element(By.NAME, "name").send_keys(name)
    driver.find_element(By.CSS_SELECTOR, "#f button").click()


def list_items(driver):
    # One read of the list's text: no <li> handles that a re-render can make stale.
    return driver.find_element(By.ID, "list").text.splitlines()


def test_add_user_flaky(driver, base):
    urllib.request.urlopen(urllib.request.Request(f"{base}/api/users", method="DELETE"))
    add_user(driver, base, "Ada")
    WebDriverWait(driver, 5).until(lambda d: list_items(d) == ["Ada"])


def test_add_user_fixed(driver, base):
    name = f"Ada {uuid.uuid4().hex[:8]}"
    add_user(driver, base, name)
    WebDriverWait(driver, 5).until(lambda d: name in list_items(d))
.venv/bin/pytest -p no:randomly test_users.py --count 20          # one process
.venv/bin/pytest -p no:randomly test_users.py --count 20 -n 4     # four processes

The flaky test passed 20 of 20 in one process and 2 of 20 with -n 4; the fixed test passed 20 of 20 both ways. (list_items reads the list in one call because of the stale element problem from the previous chapter.)

Accounts, sessions and other shared things#

Shared state is not only rows in a table. Look for anything two tests can reach at the same time:

  • One test account. Two tests logged in as the same user can log each other out, fill the same cart or change the same profile. Give each worker its own account (Playwright's testInfo.parallelIndex is a stable index from 0 to workers-1; pytest-xdist exposes worker_id), or create a fresh account per test through an API.
  • Settings and feature flags. A test that switches a global setting changes every test running at the same time. The next chapter has an example.
  • Files and ports. Two tests writing downloads/report.csv, or two dev servers trying to bind the same port. Use per-test output folders (testInfo.outputPath() in Playwright, tmp_path in pytest).
  • Counters and sequences. "The order number is 1001" only holds if nobody else placed an order. Assert on the order the test placed, found by a value the test chose.
  • External services. A sandbox payment provider or email inbox shared by the whole team. Use unique references (an email address per test, user+<id>@example.com) and filter on them.

Ten tests log in as qa@example.com. Since the suite started running with four workers, "logged out unexpectedly" failures appear. Which change removes the cause rather than the symptom?

The tests share one user, so one test's login or logout invalidates another's session. Separate accounts remove the interaction; retries and waits only make the collision less visible.

Creating data fast#

Unique data per test means more data creation. Doing it through the UI is slow and is itself a source of flakiness. Prefer an API or a database seed in a fixture, and keep the UI for the behaviour under test:

test('adds a user via the API, checks it in the UI', async ({ page, request }, testInfo) => {
  const name = `Grace ${testInfo.testId}-${testInfo.repeatEachIndex}`;
  await request.post('/api/users', { data: name });
  await page.goto('/users');
  await expect(page.getByRole('listitem').filter({ hasText: name })).toHaveCount(1);
});

Clean up what the test created, by its unique key, in an afterEach or a fixture teardown. Never clean up by deleting everything. If cleanup is expensive, let the environment be reset between CI runs instead, and make sure no test depends on an empty starting state.

Checklist#

  • Run the suite with as many workers locally as CI uses, with --repeat-each, before calling a test stable.
  • Every test creates the data it needs, with a unique key.
  • Assertions are scoped to the test's own data: no "exactly N rows" on a shared table.
  • No global resets while other tests may be running.
  • One account per worker or per test, never one for the whole suite.