---
title: 'Root Cause 4: When the App Is the Flaky Part | TestingBot'
description: Sometimes the test is right and the product is wrong some of the time.
  Race conditions, out-of-order network responses, double submits and animations that…
source_url:
  html: https://testingbot.com/resources/courses/flaky-tests/when-the-app-is-flaky
  md: https://testingbot.com/resources/courses/flaky-tests/when-the-app-is-flaky.md
---

Course

[Flaky Tests: Find, Fix and Prevent](https://testingbot.com/resources/courses/flaky-tests)

Chapter 6 of 10

1. [1 Flaky Tests: How to Find, Fix and Prevent Them](https://testingbot.com/resources/courses/flaky-tests)
2. [2 Measure Flakiness Before You Fix It](https://testingbot.com/resources/courses/flaky-tests/measuring-flakiness)
3. [3 Root Cause 1: Timing, Sleeps and Waits](https://testingbot.com/resources/courses/flaky-tests/timing-and-waits)
4. [4 Root Cause 2: Test Data and Shared State](https://testingbot.com/resources/courses/flaky-tests/test-data-and-shared-state)
5. [5 Root Cause 3: Test Order and Isolation](https://testingbot.com/resources/courses/flaky-tests/test-order-and-isolation)
6. [6 Root Cause 4: When the App Is the Flaky Part](https://testingbot.com/resources/courses/flaky-tests/when-the-app-is-flaky)
7. [7 Retries and Quarantine, Done Right](https://testingbot.com/resources/courses/flaky-tests/retries-and-quarantine)
8. [8 Debugging Flaky Tests on a Cloud Grid](https://testingbot.com/resources/courses/flaky-tests/debugging-flaky-tests-on-a-cloud-grid)
9. [9 A Team Process for Flaky Tests](https://testingbot.com/resources/courses/flaky-tests/a-team-process-for-flaky-tests)
10. [10 Flaky Tests Knowledge Check](https://testingbot.com/resources/courses/flaky-tests/knowledge-check)

[Exit course](https://testingbot.com/resources/courses)

 Chapter 6 of 10 Intermediate 9 min read 2 quick checks Updated Nov 2026 

# Root Cause 4: When the App Is the Flaky Part
On this page
1. [A reproducible example](https://testingbot.com#a-reproducible-example)
2. [Why the test caught it only 4% of the time](https://testingbot.com#why-the-test-caught-it-only-4-of-the-time)
3. [Turn the race into a deterministic failure](https://testingbot.com#turn-the-race-into-a-deterministic-failure)
4. [The fix belongs in the app](https://testingbot.com#the-fix-belongs-in-the-app)
5. [Other app-side causes worth knowing](https://testingbot.com#other-app-side-causes-worth-knowing)
6. [How to tell which side is flaky](https://testingbot.com#how-to-tell-which-side-is-flaky)

Sometimes the test is right and the product is wrong some of the time. Race conditions, out-of-order network responses, double submits and animations that swallow clicks are real bugs that real users hit, just not on every visit. A flaky test is often the first and only signal of such a bug, and the usual reflexes (add a retry, add a wait, quarantine it) throw that signal away.

## A reproducible example[#](https://testingbot.com#a-reproducible-example)

The demo app's `/search` page sends a request on every keystroke and renders the results when the response arrives. The API takes between 0 and 1000 ms. The test, in `tests/search.spec.ts`:

```typescript
import { test, expect } from '@playwright/test';

test('search narrows the results as you type', async ({ page }) => {
  await page.goto('/search');
  await page.getByLabel('Search').pressSequentially('pup');
  await expect(page.getByRole('listitem')).toHaveText(['puppeteer']);
});
```

`pressSequentially` types one character at a time, like a user, so the page sends three requests: `p`, `pu` and `pup`. Over 50 runs it passed **48 of 50**. A 4% flake with a retrying assertion looks like a test problem, and the obvious next step is a retry. Look at a failure first:

```
Expected: ["puppeteer"]
Received: ["playwright", "puppeteer", "pytest"]
```

That is the result list for `p`, not `pup`. The response for the first keystroke arrived after the responses for the later ones, and the page rendered it last. A user typing quickly on a slow connection sees results for the wrong query. The test found a real bug.

## Why the test caught it only 4% of the time[#](https://testingbot.com#why-the-test-caught-it-only-4-of-the-time)

The bug happens whenever the `p` response is the last to arrive, which with random latencies is roughly one time in three. The test failed far less often, because `toHaveText` passes the moment it sees `["puppeteer"]`. When `pu` came back first, the list briefly showed the right result, the assertion passed, and the stale `p` results replaced it a few hundred milliseconds later, after the test had finished.

A passing retrying assertion proves the page reached that state once, not that it stayed there. For a page that is still working in the background, wait until it says it is done. The demo page marks its result list with `aria-busy` while requests are in flight, so the test can wait for the page to settle first:

```typescript
test('search shows the right results once it is idle', async ({ page }) => {
  await page.goto('/search');
  await page.getByLabel('Search').pressSequentially('pup');
  await expect(page.getByRole('list')).toHaveAttribute('aria-busy', 'false');
  await expect(page.getByRole('listitem')).toHaveText(['puppeteer']);
});
```

This version passed **33 of 50** : it now catches the bug about as often as the bug happens. A test that fails more often is not a worse test here; it is a more honest one. If your app has no busy signal, adding one (an `aria-busy` attribute, a loading spinner that disappears, a `data-state` attribute) helps both tests and users of assistive technology.

A search test with `await expect(results).toHaveText(['puppeteer'])` passes, yet a user reports seeing results for an older query. How can both be true?

 The assertion timeout was too long `toHaveText` does not wait for network responses at all The assertion passed while the right results were briefly shown, and a late response replaced them afterwards The test ran in a fresh browser context

A retrying assertion stops at the first moment the condition is true. If the page is still processing responses, it can change after the test has moved on. Wait for a "done" signal before asserting the final state.

## Turn the race into a deterministic failure[#](https://testingbot.com#turn-the-race-into-a-deterministic-failure)

A 30% failure is better than 4%, but you can do better still: force the order you suspect. Playwright's `page.route` can hold back one response until the others have been answered:

```typescript
test('a slow first response does not win', async ({ page }) => {
  // Force the suspected order: "p" is answered after "pu" and "pup".
  const answered = new Set<string>();
  let release!: () => void;
  const laterQueriesDone = new Promise<void>((resolve) => (release = resolve));
  page.on('requestfinished', (r) => {
    answered.add(new URL(r.url()).searchParams.get('q') ?? '');
    if (answered.has('pu') && answered.has('pup')) release();
  });
  await page.route('**/api/search?q=p', async (route) => {
    await laterQueriesDone;
    await route.continue();
  });

  await page.goto('/search');
  await page.getByLabel('Search').pressSequentially('pup');
  await expect(page.getByRole('list')).toHaveAttribute('aria-busy', 'false');
  await expect(page.getByRole('listitem')).toHaveText(['puppeteer']);
  await page.unrouteAll({ behavior: 'ignoreErrors' });
});
```

Against `/search` it failed **50 of 50**. The flake is now a bug report with a reliable reproduction, which is what a developer needs to fix it. The test waits on events, not on time: the held request is released when the other two have finished, however long that takes.

This is the strongest way to tell "the app is flaky" from "the test is flaky": if you can force the suspected order or timing and the failure becomes certain, the product has the bug. Useful levers:

- **Network order and delay** : `page.route` (Playwright) to delay, reorder or fail specific requests.
- **CPU and network speed** : Chrome DevTools throttling, or `page.clock` in Playwright to control timers.
- **Repetition under load** : run the test with many workers to slow everything down; races that depend on timing show up more often.

In Selenium, request interception is more limited: Chrome's DevTools Protocol (through `driver.execute_cdp_cmd`) can throttle the network, and a test proxy can delay specific requests. Often the simpler route is to reproduce the race in the app's own unit or component tests, where you control the fetch function directly.

## The fix belongs in the app[#](https://testingbot.com#the-fix-belongs-in-the-app)

The page needs to ignore responses for queries that are no longer current. The demo app gets a second page, `/search-fixed`, with one change: each keystroke cancels the previous request. Add it to the `pages` object in `app/server.mjs`:

```javascript
// 4b. The same page with the bug fixed: a new keystroke cancels the previous request.
  '/search-fixed': `<label>Search <input id="q"></label><ul id="results" aria-busy="false"></ul>
<script>
const list = document.getElementById('results');
let pending = 0;
let controller;
document.getElementById('q').oninput = async (e) => {
  controller?.abort(); // the fix: a new query cancels the old one
  controller = new AbortController();
  list.ariaBusy = String(++pending > 0);
  try {
    const res = await fetch('/api/search?q=' + encodeURIComponent(e.target.value),
                            { signal: controller.signal });
    const items = await res.json();
    list.replaceChildren(...items.map((i) => {
    const li = document.createElement('li');
    li.textContent = i;
    return li;
  }));
  } catch (err) {
    if (err.name !== 'AbortError') throw err;
  } finally {
    list.ariaBusy = String(--pending > 0);
  }
};
</script>`,
```

Run all three tests against both pages with `--repeat-each 50 --workers 4`. To do that, wrap each test in a loop and use the path in its title and `goto`:

```typescript
for (const path of ['/search', '/search-fixed']) {
  test(`search shows the right results once it is idle ${path}`, async ({ page }) => {
    await page.goto(path);
    // ... same steps as above
  });
}
```

| Test | `/search` | `/search-fixed` |
| --- | --- | --- |
| narrows the results as you type | 48/50 | 50/50 |
| right results once it is idle | 33/50 | 50/50 |
| a slow first response does not win | 0/50 | 50/50 |

Keep the deterministic test in the suite. It is a regression test for the race, and unlike the original it will fail every time if someone removes the `abort()`.

You suspect a flaky checkout test is caught by a real double-submit bug. What is the most convincing next step?

 Add `retries: 3` and see if it passes Wrap the click in a try/catch Ask the AI assistant whether it is a test problem Force the suspected condition (for example delay the first submit response with `page.route`) and check whether the failure becomes certain

If forcing the suspected timing makes the failure reproducible every time, the app has the bug and you have a regression test for it. Retries and try/catch only hide the signal.

## Other app-side causes worth knowing[#](https://testingbot.com#other-app-side-causes-worth-knowing)

- **Animations and transitions.** A button that slides in can receive a click at a position where it no longer is. Playwright waits for an element to be stable (not moving) before clicking; Selenium does not, and reports `ElementClickInterceptedException` or clicks the wrong thing. For tests, disabling animations is legitimate (Playwright's `toHaveScreenshot` does it by default; many apps honour `prefers-reduced-motion`, which `page.emulateMedia({ reducedMotion: 'reduce' })` sets). If real users can click during the animation, though, the app should handle it.
- **Hydration.** A server-rendered page shows a button before its JavaScript is attached. A fast test clicks it and nothing happens. Users can do the same. Wait for a signal that the page is interactive, and consider whether the app should disable controls until it is.
- **Optimistic UI and eventual consistency.** The page shows "Saved" before the server confirmed, or a list reads from a replica that lags behind. The test sees the old data some of the time; so do users.
- **Time.** Code that breaks around midnight, at month end, or in another time zone fails on the CI runner that happens to cross the boundary. Control the clock (`page.clock` in Playwright) and the time zone (a `timezoneId` per project) in tests that depend on dates.

## How to tell which side is flaky[#](https://testingbot.com#how-to-tell-which-side-is-flaky)

| Signal | Points at the test | Points at the app |
| --- | --- | --- |
| The failure value | "element not found", empty text, timeouts at a fixed sleep | A wrong but plausible value: old results, duplicate rows, a stale total |
| Forcing the timing | Failure stays random | Failure becomes certain |
| Manual check | Cannot reproduce by hand | Reproducible by hand on a slow network |
| Other tests | Only this test | Other tests of the same feature also flake |

When the evidence points at the app, file the bug with the deterministic reproduction, mark the test as an expected failure linked to the bug (chapter 7), and do not "fix" the test by loosening it.

[Previous · Chapter 5 Root Cause 3: Test Order and Isolation](https://testingbot.com/resources/courses/flaky-tests/test-order-and-isolation) [Next · Chapter 7 Retries and Quarantine, Done Right](https://testingbot.com/resources/courses/flaky-tests/retries-and-quarantine)

On this page

1. [A reproducible example](https://testingbot.com#a-reproducible-example)
2. [Why the test caught it only 4% of the time](https://testingbot.com#why-the-test-caught-it-only-4-of-the-time)
3. [Turn the race into a deterministic failure](https://testingbot.com#turn-the-race-into-a-deterministic-failure)
4. [The fix belongs in the app](https://testingbot.com#the-fix-belongs-in-the-app)
5. [Other app-side causes worth knowing](https://testingbot.com#other-app-side-causes-worth-knowing)
6. [How to tell which side is flaky](https://testingbot.com#how-to-tell-which-side-is-flaky)

Certification

Earn the free, resume-worthy TestingBot Flaky Test Fixing Certification for this course.

[How it works](https://testingbot.com/resources/courses/flaky-tests/certification)
