---
title: Debugging Flaky Tests on a Cloud Grid | TestingBot
description: Many flaky tests only fail in CI. The CI machine is slower or faster
  than your laptop, runs a different operating system, and talks to the browser over
  a…
source_url:
  html: https://testingbot.com/resources/courses/flaky-tests/debugging-flaky-tests-on-a-cloud-grid
  md: https://testingbot.com/resources/courses/flaky-tests/debugging-flaky-tests-on-a-cloud-grid.md
---

Course

[Flaky Tests: Find, Fix and Prevent](https://testingbot.com/resources/courses/flaky-tests)

Chapter 8 of 10

1. [1 Flaky Tests: How to Find, Fix and Prevent Them](https://testingbot.com/resources/courses/flaky-tests)
2. [2 Measure Flakiness Before You Fix It](https://testingbot.com/resources/courses/flaky-tests/measuring-flakiness)
3. [3 Root Cause 1: Timing, Sleeps and Waits](https://testingbot.com/resources/courses/flaky-tests/timing-and-waits)
4. [4 Root Cause 2: Test Data and Shared State](https://testingbot.com/resources/courses/flaky-tests/test-data-and-shared-state)
5. [5 Root Cause 3: Test Order and Isolation](https://testingbot.com/resources/courses/flaky-tests/test-order-and-isolation)
6. [6 Root Cause 4: When the App Is the Flaky Part](https://testingbot.com/resources/courses/flaky-tests/when-the-app-is-flaky)
7. [7 Retries and Quarantine, Done Right](https://testingbot.com/resources/courses/flaky-tests/retries-and-quarantine)
8. [8 Debugging Flaky Tests on a Cloud Grid](https://testingbot.com/resources/courses/flaky-tests/debugging-flaky-tests-on-a-cloud-grid)
9. [9 A Team Process for Flaky Tests](https://testingbot.com/resources/courses/flaky-tests/a-team-process-for-flaky-tests)
10. [10 Flaky Tests Knowledge Check](https://testingbot.com/resources/courses/flaky-tests/knowledge-check)

[Exit course](https://testingbot.com/resources/courses)

 Chapter 8 of 10 Intermediate 9 min read 2 quick checks Updated Nov 2026 

# Debugging Flaky Tests on a Cloud Grid
On this page
1. [Run the local app on remote browsers](https://testingbot.com#run-the-local-app-on-remote-browsers)
2. [A conftest that can target the grid](https://testingbot.com#a-conftest-that-can-target-the-grid)
3. [Nine runs of the flaky test](https://testingbot.com#nine-runs-of-the-flaky-test)
4. [Compare a pass with a fail](https://testingbot.com#compare-a-pass-with-a-fail)
5. [Using the history](https://testingbot.com#using-the-history)
6. [Infrastructure failures look different](https://testingbot.com#infrastructure-failures-look-different)
7. [Checklist](https://testingbot.com#checklist)

Many flaky tests only fail in CI. The CI machine is slower or faster than your laptop, runs a different operating system, and talks to the browser over a network. A cloud grid adds more of the same: browsers on Windows and macOS, real network latency between the test and the browser, and many runs on many machines. That makes it a good place to _find_ flakiness, and, if each run leaves a video, logs and a step list behind, a good place to debug it.

This chapter runs the flaky save test from chapter 3 on TestingBot, nine times, and compares a passing run with a failing one.

## Run the local app on remote browsers[#](https://testingbot.com#run-the-local-app-on-remote-browsers)

The demo app runs on your machine, which a remote browser cannot reach directly. TestingBot Tunnel opens a secure connection from your machine to the grid. Start it with an identifier:

```bash
java -jar testingbot-tunnel.jar $TB_KEY $TB_SECRET --tunnel-identifier flaky-course
```

and keep it running. Remote browsers then load the app at `http://local.testingbot.com:4817`, a hostname that points to `127.0.0.1` and, through the tunnel, at your machine. The [local website testing course](https://testingbot.com/resources/courses/local-website-testing) explains the tunnel in depth.

## A conftest that can target the grid[#](https://testingbot.com#a-conftest-that-can-target-the-grid)

Extend `py/conftest.py` from chapter 2 with a `--grid` option. Three details matter for flaky tests:

```python
import os

import pytest
from selenium import webdriver

BASE = os.environ.get("BASE_URL", "http://localhost:4817")
GRID_URL = "https://hub.testingbot.com/wd/hub"
BROWSERS = {"chrome": webdriver.ChromeOptions, "firefox": webdriver.FirefoxOptions}

def pytest_addoption(parser):
    parser.addoption("--grid", action="store_true", help="run on the TestingBot grid")
    parser.addoption("--browser", default="chrome", choices=sorted(BROWSERS))

@pytest.hookimpl(wrapper=True, tryfirst=True)
def pytest_runtest_makereport(item, call):
    report = yield
    setattr(item, "rep_" + report.when, report)
    return report

@pytest.fixture
def base():
    return BASE

@pytest.fixture
def driver(request):
    config = request.config
    browser = config.getoption("browser")
    options = BROWSERS[browser]()
    if config.getoption("grid"):
        options.browser_version = "latest"
        options.platform_name = os.environ.get("TB_PLATFORM", "WIN11")
        options.set_capability("tb:options", {
            "key": os.environ["TB_KEY"],
            "secret": os.environ["TB_SECRET"],
            # One stable name per test, so every run lands in the same history.
            "name": f"{request.node.module. __name__ }::{request.node.originalname}",
            "build": os.environ.get("TB_BUILD", "local"),
            "tunnel-identifier": os.environ.get("TB_TUNNEL"),
        })
        d = webdriver.Remote(command_executor=GRID_URL, options=options)
    else:
        options.add_argument("--headless=new" if browser == "chrome" else "-headless")
        d = (webdriver.Chrome if browser == "chrome" else webdriver.Firefox)(options=options)

    yield d

    call = getattr(request.node, "rep_call", None)
    try:
        if config.getoption("grid"):
            passed = call is not None and call.passed
            d.execute_script(f"tb:test-result={'passed' if passed else 'failed'}")
            print(f"\nTestingBot session {d.session_id}: {'passed' if passed else 'failed'}")
    finally:
        d.quit()
```

1. **A stable test name.** `originalname` drops pytest's parameter suffix, so `test_save_flaky[3-20]` from pytest-repeat and a plain run both report as `test_timing::test_save_flaky`. History, flips and the flaky label all group runs by exact name.
2. **A build name.** `TB_BUILD` groups the sessions of one CI run, so a build page can compare it with the build before.
3. **A reported result.** The grid cannot see your assertions. The `tb:test-result` command marks the session passed or failed before `quit()`; without it, every run counts as unknown and cannot be flaky or failing. The [annotating tests guide](https://testingbot.com/support/web-automate/selenium/annotating-tests) lists the other `tb:` commands.

For Playwright, the course's [cloud grid chapter](https://testingbot.com/resources/courses/playwright/run-on-a-cloud-grid) has the equivalent: a `connectOptions.wsEndpoint` per project and an automatic fixture that sends `setSessionName` and `setSessionStatus`. Note that a Playwright session on TestingBot follows a worker, not a test, so per-test history works best with the Selenium approach above or with one test per worker.

## Nine runs of the flaky test[#](https://testingbot.com#nine-runs-of-the-flaky-test)

One at a time, so the runs do not compete for parallel slots:

```bash
export TB_KEY=... TB_SECRET=... TB_TUNNEL=flaky-course TB_BUILD=flaky-tests-course
export BASE_URL=http://local.testingbot.com:4817
for i in $(seq 1 9); do
  .venv/bin/pytest -p no:randomly -s test_timing.py -k flaky --grid | grep "TestingBot session"
done
```

Five runs passed and four failed, on Chrome 156 on Windows 11. The last eight, oldest first:

| Session | Result |
| --- | --- |
| `ab41b0f09844-e0e0a032ca4a-1f6f393442d9-179449890511-0887463` | passed |
| `53c6dc908b8f-e32af960e165-dfb9a04c067f-179449891325-16854918` | passed |
| `8adbad83a19b-0eb3ba2932c0-feb83ee3a4a1-179449892255-91431287` | passed |
| `b1d7adebf18d-b0850062dec2-2d1bf9287f83-179449893189-95571638` | failed |
| `3c1f3b1d243b-4a19d9334d77-cca1240d5a3f-179449894174-01688997` | passed |
| `547952f9eccc-a23a86186d04-a2b848b34777-179449895063-13498530` | passed |
| `db66258fc814-37b4572d0e1a-11ccb0aa42f8-179449895982-28728772` | failed |
| `c441bc845891-6b9489ca425c-3c991d39c2fc-179449896844-46824205` | failed |

That is 4 failed runs out of 9 with 4 flips, which meets the documented rule for the **Flaky** label on a test's [All runs](https://testingbot.com/support/analytics/insights) page: on one environment, at least 5 decided runs, between 10% and 90% failed, and at least two flips. The [Insights API](https://testingbot.com/support/api/insights) agreed. Its per-test endpoint returned this entry:

```bash
curl -u "$TB_KEY:$TB_SECRET" "https://api.testingbot.com/v1/insights/test-cases?time_range=7d&sort=flaky&count=50"
```

```json
{"name": "test_timing::test_save_flaky", "total": 9, "success": 5, "failure": 4,
 "unknown": 0, "pass_rate": 0.5556, "flaky": true, "avg_duration": 1.89}
```

## Compare a pass with a fail[#](https://testingbot.com#compare-a-pass-with-a-fail)

Open one passing and one failing session side by side. On a test's detail page, the **Steps** tab lists every WebDriver command with its duration and response, and clicking a step moves the video to that moment. For these two runs the steps were identical up to the end:

| Step | Passed run (`547952f9...`) | Failed run (`c441bc84...`) |
| --- | --- | --- |
| Open URL `/save` | 328 ms | 481 ms |
| Find Element `#save` | 77 ms | 72 ms |
| Click Element | 161 ms | 165 ms |
| Find Element `#status` | 64 ms + 0.5s idle | 65 ms + 0.5s idle |
| Get Text From Element | `"Saved"` | `""` |

Three things stand out, and they are the same three you look for in any flaky failure:

1. **The divergence point.** Everything matched until the text was read. The bug is in what happens between the click and the read, not in finding elements.
2. **The idle time.** The step list shows the test doing nothing for 0.5 s before reading the status: the `time.sleep(0.5)`. A fixed pause in front of the step that diverges is the fingerprint of a timing problem.
3. **The value, not just the error.** The failed run read an empty string, not a wrong one. Empty means "not yet"; a wrong but plausible value (old search results, a duplicate row) would point at shared data or at the app, as in chapters 4 and 6.

The video adds what the step list cannot: what the page looked like at that moment, including spinners, overlays and animations. The **Logs** tab has the Selenium log and the browser's log for the session, where a JavaScript error or a failed request on the page often explains a "random" failure.

## Using the history[#](https://testingbot.com#using-the-history)

A test's detail page shows a History strip of its last 20 runs, and **All runs** opens every run of the test with:

- a verdict sentence such as "Flaky on Chrome 156: ..." naming where it flakes,
- the pass rate, number of runs and **flips** (how often the result changed between consecutive runs on the same environment),
- a breakdown **By environment** , which tells you whether the test flakes everywhere or only on one browser,
- **Why it failed** , which groups the failures by error message and the step they failed at.

These answer the questions that decide the fix. Flaky only on Safari points at a browser-specific timing or rendering difference. Flaky everywhere since one build points at a change in that build. Failures that all share one error at one step point at one cause; several different errors point at an environment problem or several causes. The [Test Analytics guide](https://testingbot.com/support/analytics) walks through a full triage, and [build analytics](https://testingbot.com/support/analytics/builds) show, per build, which failures are new and which tests were already flaky in the builds before, so a flaky test is not blamed on the change under review.

A test's All runs page shows a 100% pass rate on Chrome and Firefox and 65% on Safari, with every failure grouped under one error at the same step. Where do you start?

 Add retries for all browsers Compare a passing and a failing Safari run at that step: video, step timings and the value returned Drop Safari from the test matrix Raise the global timeout

One browser and one error at one step is a narrow, debuggable signal. Comparing a pass with a fail on that browser shows where the runs diverge; dropping the browser or retrying would discard exactly the evidence that locates the bug.

## Infrastructure failures look different[#](https://testingbot.com#infrastructure-failures-look-different)

Not every failure on a grid is the test's fault. Infrastructure problems tend to have a different shape:

- The session never starts, or fails at "Start Session", rather than at an assertion.
- The browser disconnects in the middle of an action, with a connection error instead of an assertion or a timeout on a locator.
- Different tests fail on different runs, each once, rather than the same test repeatedly.

The first two are worth a narrow retry of the session start (chapter 7). If one test fails at the same step with the same error again and again, it is not the infrastructure.

AI Insights, if your account owner has enabled it, adds an explanation of a failed run with a "likely owner" (application, test script, environment or flaky) and a confidence score. The [AI Insights documentation](https://testingbot.com/support/ai/insights) is explicit that it is less certain on ambiguous cases such as flaky timing, so treat it as a second opinion next to the comparison above, not as a verdict.

Your tests report a result to TestingBot, but the All runs page never labels anything flaky, even for a test you know fails half the time. What is the most likely reason?

 The test runs on too many browsers Flaky labels need retries to be enabled The test name changes between runs, for example it includes a parameter, timestamp or build number Videos are turned off

History groups runs by exact name. If each run has its own name, each "test" has a single run and nothing can be flaky. Keep names stable and put run-specific values in the build.

## Checklist[#](https://testingbot.com#checklist)

- Every grid session has a stable name, a build and a reported result.
- When a test flakes, open one passing and one failing run and find the first step where they differ.
- Read the value at that step: empty or missing means timing, wrong but plausible means data or the app.
- Use the environment breakdown and grouped errors before touching the code.
- Retry only failures that happened before your test did anything.

[Previous · Chapter 7 Retries and Quarantine, Done Right](https://testingbot.com/resources/courses/flaky-tests/retries-and-quarantine) [Next · Chapter 9 A Team Process for Flaky Tests](https://testingbot.com/resources/courses/flaky-tests/a-team-process-for-flaky-tests)

On this page

1. [Run the local app on remote browsers](https://testingbot.com#run-the-local-app-on-remote-browsers)
2. [A conftest that can target the grid](https://testingbot.com#a-conftest-that-can-target-the-grid)
3. [Nine runs of the flaky test](https://testingbot.com#nine-runs-of-the-flaky-test)
4. [Compare a pass with a fail](https://testingbot.com#compare-a-pass-with-a-fail)
5. [Using the history](https://testingbot.com#using-the-history)
6. [Infrastructure failures look different](https://testingbot.com#infrastructure-failures-look-different)
7. [Checklist](https://testingbot.com#checklist)

Certification

Earn the free, resume-worthy TestingBot Flaky Test Fixing Certification for this course.

[How it works](https://testingbot.com/resources/courses/flaky-tests/certification)
