Root Cause 4: When the App Is the Flaky Part
On this page
Sometimes the test is right and the product is wrong some of the time. Race conditions, out-of-order network responses, double submits and animations that swallow clicks are real bugs that real users hit, just not on every visit. A flaky test is often the first and only signal of such a bug, and the usual reflexes (add a retry, add a wait, quarantine it) throw that signal away.
A reproducible example#
The demo app's /search page sends a request on every keystroke and renders the results when the response arrives. The API takes between 0 and 1000 ms. The test, in tests/search.spec.ts:
import { test, expect } from '@playwright/test';
test('search narrows the results as you type', async ({ page }) => {
await page.goto('/search');
await page.getByLabel('Search').pressSequentially('pup');
await expect(page.getByRole('listitem')).toHaveText(['puppeteer']);
});
pressSequentially types one character at a time, like a user, so the page sends three requests: p, pu and pup. Over 50 runs it passed 48 of 50. A 4% flake with a retrying assertion looks like a test problem, and the obvious next step is a retry. Look at a failure first:
Expected: ["puppeteer"]
Received: ["playwright", "puppeteer", "pytest"]
That is the result list for p, not pup. The response for the first keystroke arrived after the responses for the later ones, and the page rendered it last. A user typing quickly on a slow connection sees results for the wrong query. The test found a real bug.
Why the test caught it only 4% of the time#
The bug happens whenever the p response is the last to arrive, which with random latencies is roughly one time in three. The test failed far less often, because toHaveText passes the moment it sees ["puppeteer"]. When pu came back first, the list briefly showed the right result, the assertion passed, and the stale p results replaced it a few hundred milliseconds later, after the test had finished.
A passing retrying assertion proves the page reached that state once, not that it stayed there. For a page that is still working in the background, wait until it says it is done. The demo page marks its result list with aria-busy while requests are in flight, so the test can wait for the page to settle first:
test('search shows the right results once it is idle', async ({ page }) => {
await page.goto('/search');
await page.getByLabel('Search').pressSequentially('pup');
await expect(page.getByRole('list')).toHaveAttribute('aria-busy', 'false');
await expect(page.getByRole('listitem')).toHaveText(['puppeteer']);
});
This version passed 33 of 50: it now catches the bug about as often as the bug happens. A test that fails more often is not a worse test here; it is a more honest one. If your app has no busy signal, adding one (an aria-busy attribute, a loading spinner that disappears, a data-state attribute) helps both tests and users of assistive technology.
A search test with await expect(results).toHaveText(['puppeteer']) passes, yet a user reports seeing results for an older query. How can both be true?
A retrying assertion stops at the first moment the condition is true. If the page is still processing responses, it can change after the test has moved on. Wait for a "done" signal before asserting the final state.
Turn the race into a deterministic failure#
A 30% failure is better than 4%, but you can do better still: force the order you suspect. Playwright's page.route can hold back one response until the others have been answered:
test('a slow first response does not win', async ({ page }) => {
// Force the suspected order: "p" is answered after "pu" and "pup".
const answered = new Set<string>();
let release!: () => void;
const laterQueriesDone = new Promise<void>((resolve) => (release = resolve));
page.on('requestfinished', (r) => {
answered.add(new URL(r.url()).searchParams.get('q') ?? '');
if (answered.has('pu') && answered.has('pup')) release();
});
await page.route('**/api/search?q=p', async (route) => {
await laterQueriesDone;
await route.continue();
});
await page.goto('/search');
await page.getByLabel('Search').pressSequentially('pup');
await expect(page.getByRole('list')).toHaveAttribute('aria-busy', 'false');
await expect(page.getByRole('listitem')).toHaveText(['puppeteer']);
await page.unrouteAll({ behavior: 'ignoreErrors' });
});
Against /search it failed 50 of 50. The flake is now a bug report with a reliable reproduction, which is what a developer needs to fix it. The test waits on events, not on time: the held request is released when the other two have finished, however long that takes.
This is the strongest way to tell "the app is flaky" from "the test is flaky": if you can force the suspected order or timing and the failure becomes certain, the product has the bug. Useful levers:
-
Network order and delay:
page.route(Playwright) to delay, reorder or fail specific requests. -
CPU and network speed: Chrome DevTools throttling, or
page.clockin Playwright to control timers. - Repetition under load: run the test with many workers to slow everything down; races that depend on timing show up more often.
In Selenium, request interception is more limited: Chrome's DevTools Protocol (through driver.execute_cdp_cmd) can throttle the network, and a test proxy can delay specific requests. Often the simpler route is to reproduce the race in the app's own unit or component tests, where you control the fetch function directly.
The fix belongs in the app#
The page needs to ignore responses for queries that are no longer current. The demo app gets a second page, /search-fixed, with one change: each keystroke cancels the previous request. Add it to the pages object in app/server.mjs:
// 4b. The same page with the bug fixed: a new keystroke cancels the previous request.
'/search-fixed': `<label>Search <input id="q"></label><ul id="results" aria-busy="false"></ul>
<script>
const list = document.getElementById('results');
let pending = 0;
let controller;
document.getElementById('q').oninput = async (e) => {
controller?.abort(); // the fix: a new query cancels the old one
controller = new AbortController();
list.ariaBusy = String(++pending > 0);
try {
const res = await fetch('/api/search?q=' + encodeURIComponent(e.target.value),
{ signal: controller.signal });
const items = await res.json();
list.replaceChildren(...items.map((i) => {
const li = document.createElement('li');
li.textContent = i;
return li;
}));
} catch (err) {
if (err.name !== 'AbortError') throw err;
} finally {
list.ariaBusy = String(--pending > 0);
}
};
</script>`,
Run all three tests against both pages with --repeat-each 50 --workers 4. To do that, wrap each test in a loop and use the path in its title and goto:
for (const path of ['/search', '/search-fixed']) {
test(`search shows the right results once it is idle ${path}`, async ({ page }) => {
await page.goto(path);
// ... same steps as above
});
}
| Test | /search |
/search-fixed |
|---|---|---|
| narrows the results as you type | 48/50 | 50/50 |
| right results once it is idle | 33/50 | 50/50 |
| a slow first response does not win | 0/50 | 50/50 |
Keep the deterministic test in the suite. It is a regression test for the race, and unlike the original it will fail every time if someone removes the abort().
You suspect a flaky checkout test is caught by a real double-submit bug. What is the most convincing next step?
If forcing the suspected timing makes the failure reproducible every time, the app has the bug and you have a regression test for it. Retries and try/catch only hide the signal.
Other app-side causes worth knowing#
-
Animations and transitions. A button that slides in can receive a click at a position where it no longer is. Playwright waits for an element to be stable (not moving) before clicking; Selenium does not, and reports
ElementClickInterceptedExceptionor clicks the wrong thing. For tests, disabling animations is legitimate (Playwright'stoHaveScreenshotdoes it by default; many apps honourprefers-reduced-motion, whichpage.emulateMedia({ reducedMotion: 'reduce' })sets). If real users can click during the animation, though, the app should handle it. - Hydration. A server-rendered page shows a button before its JavaScript is attached. A fast test clicks it and nothing happens. Users can do the same. Wait for a signal that the page is interactive, and consider whether the app should disable controls until it is.
- Optimistic UI and eventual consistency. The page shows "Saved" before the server confirmed, or a list reads from a replica that lags behind. The test sees the old data some of the time; so do users.
-
Time. Code that breaks around midnight, at month end, or in another time zone fails on the CI runner that happens to cross the boundary. Control the clock (
page.clockin Playwright) and the time zone (atimezoneIdper project) in tests that depend on dates.
How to tell which side is flaky#
| Signal | Points at the test | Points at the app |
|---|---|---|
| The failure value | "element not found", empty text, timeouts at a fixed sleep | A wrong but plausible value: old results, duplicate rows, a stale total |
| Forcing the timing | Failure stays random | Failure becomes certain |
| Manual check | Cannot reproduce by hand | Reproducible by hand on a slow network |
| Other tests | Only this test | Other tests of the same feature also flake |
When the evidence points at the app, file the bug with the deterministic reproduction, mark the test as an expected failure linked to the bug (chapter 7), and do not "fix" the test by loosening it.