Exit course
Chapter 8 of 10 Intermediate 9 min read 2 quick checks Updated

Debugging Flaky Tests on a Cloud Grid

On this page
  1. Run the local app on remote browsers
  2. A conftest that can target the grid
  3. Nine runs of the flaky test
  4. Compare a pass with a fail
  5. Using the history
  6. Infrastructure failures look different
  7. Checklist

Many flaky tests only fail in CI. The CI machine is slower or faster than your laptop, runs a different operating system, and talks to the browser over a network. A cloud grid adds more of the same: browsers on Windows and macOS, real network latency between the test and the browser, and many runs on many machines. That makes it a good place to find flakiness, and, if each run leaves a video, logs and a step list behind, a good place to debug it.

This chapter runs the flaky save test from chapter 3 on TestingBot, nine times, and compares a passing run with a failing one.

Run the local app on remote browsers#

The demo app runs on your machine, which a remote browser cannot reach directly. TestingBot Tunnel opens a secure connection from your machine to the grid. Start it with an identifier:

java -jar testingbot-tunnel.jar $TB_KEY $TB_SECRET --tunnel-identifier flaky-course

and keep it running. Remote browsers then load the app at http://local.testingbot.com:4817, a hostname that points to 127.0.0.1 and, through the tunnel, at your machine. The local website testing course explains the tunnel in depth.

A conftest that can target the grid#

Extend py/conftest.py from chapter 2 with a --grid option. Three details matter for flaky tests:

import os

import pytest
from selenium import webdriver

BASE = os.environ.get("BASE_URL", "http://localhost:4817")
GRID_URL = "https://hub.testingbot.com/wd/hub"
BROWSERS = {"chrome": webdriver.ChromeOptions, "firefox": webdriver.FirefoxOptions}


def pytest_addoption(parser):
    parser.addoption("--grid", action="store_true", help="run on the TestingBot grid")
    parser.addoption("--browser", default="chrome", choices=sorted(BROWSERS))


@pytest.hookimpl(wrapper=True, tryfirst=True)
def pytest_runtest_makereport(item, call):
    report = yield
    setattr(item, "rep_" + report.when, report)
    return report


@pytest.fixture
def base():
    return BASE


@pytest.fixture
def driver(request):
    config = request.config
    browser = config.getoption("browser")
    options = BROWSERS[browser]()
    if config.getoption("grid"):
        options.browser_version = "latest"
        options.platform_name = os.environ.get("TB_PLATFORM", "WIN11")
        options.set_capability("tb:options", {
            "key": os.environ["TB_KEY"],
            "secret": os.environ["TB_SECRET"],
            # One stable name per test, so every run lands in the same history.
            "name": f"{request.node.module.__name__}::{request.node.originalname}",
            "build": os.environ.get("TB_BUILD", "local"),
            "tunnel-identifier": os.environ.get("TB_TUNNEL"),
        })
        d = webdriver.Remote(command_executor=GRID_URL, options=options)
    else:
        options.add_argument("--headless=new" if browser == "chrome" else "-headless")
        d = (webdriver.Chrome if browser == "chrome" else webdriver.Firefox)(options=options)

    yield d

    call = getattr(request.node, "rep_call", None)
    try:
        if config.getoption("grid"):
            passed = call is not None and call.passed
            d.execute_script(f"tb:test-result={'passed' if passed else 'failed'}")
            print(f"\nTestingBot session {d.session_id}: {'passed' if passed else 'failed'}")
    finally:
        d.quit()
  1. A stable test name. originalname drops pytest's parameter suffix, so test_save_flaky[3-20] from pytest-repeat and a plain run both report as test_timing::test_save_flaky. History, flips and the flaky label all group runs by exact name.
  2. A build name. TB_BUILD groups the sessions of one CI run, so a build page can compare it with the build before.
  3. A reported result. The grid cannot see your assertions. The tb:test-result command marks the session passed or failed before quit(); without it, every run counts as unknown and cannot be flaky or failing. The annotating tests guide lists the other tb: commands.

For Playwright, the course's cloud grid chapter has the equivalent: a connectOptions.wsEndpoint per project and an automatic fixture that sends setSessionName and setSessionStatus. Note that a Playwright session on TestingBot follows a worker, not a test, so per-test history works best with the Selenium approach above or with one test per worker.

Nine runs of the flaky test#

One at a time, so the runs do not compete for parallel slots:

export TB_KEY=... TB_SECRET=... TB_TUNNEL=flaky-course TB_BUILD=flaky-tests-course
export BASE_URL=http://local.testingbot.com:4817
for i in $(seq 1 9); do
  .venv/bin/pytest -p no:randomly -s test_timing.py -k flaky --grid | grep "TestingBot session"
done

Five runs passed and four failed, on Chrome 156 on Windows 11. The last eight, oldest first:

Session Result
ab41b0f09844-e0e0a032ca4a-1f6f393442d9-179449890511-0887463 passed
53c6dc908b8f-e32af960e165-dfb9a04c067f-179449891325-16854918 passed
8adbad83a19b-0eb3ba2932c0-feb83ee3a4a1-179449892255-91431287 passed
b1d7adebf18d-b0850062dec2-2d1bf9287f83-179449893189-95571638 failed
3c1f3b1d243b-4a19d9334d77-cca1240d5a3f-179449894174-01688997 passed
547952f9eccc-a23a86186d04-a2b848b34777-179449895063-13498530 passed
db66258fc814-37b4572d0e1a-11ccb0aa42f8-179449895982-28728772 failed
c441bc845891-6b9489ca425c-3c991d39c2fc-179449896844-46824205 failed

That is 4 failed runs out of 9 with 4 flips, which meets the documented rule for the Flaky label on a test's All runs page: on one environment, at least 5 decided runs, between 10% and 90% failed, and at least two flips. The Insights API agreed. Its per-test endpoint returned this entry:

curl -u "$TB_KEY:$TB_SECRET" "https://api.testingbot.com/v1/insights/test-cases?time_range=7d&sort=flaky&count=50"
{"name": "test_timing::test_save_flaky", "total": 9, "success": 5, "failure": 4,
 "unknown": 0, "pass_rate": 0.5556, "flaky": true, "avg_duration": 1.89}

Compare a pass with a fail#

Open one passing and one failing session side by side. On a test's detail page, the Steps tab lists every WebDriver command with its duration and response, and clicking a step moves the video to that moment. For these two runs the steps were identical up to the end:

Step Passed run (547952f9...) Failed run (c441bc84...)
Open URL /save 328 ms 481 ms
Find Element #save 77 ms 72 ms
Click Element 161 ms 165 ms
Find Element #status 64 ms + 0.5s idle 65 ms + 0.5s idle
Get Text From Element "Saved" ""

Three things stand out, and they are the same three you look for in any flaky failure:

  1. The divergence point. Everything matched until the text was read. The bug is in what happens between the click and the read, not in finding elements.
  2. The idle time. The step list shows the test doing nothing for 0.5 s before reading the status: the time.sleep(0.5). A fixed pause in front of the step that diverges is the fingerprint of a timing problem.
  3. The value, not just the error. The failed run read an empty string, not a wrong one. Empty means "not yet"; a wrong but plausible value (old search results, a duplicate row) would point at shared data or at the app, as in chapters 4 and 6.

The video adds what the step list cannot: what the page looked like at that moment, including spinners, overlays and animations. The Logs tab has the Selenium log and the browser's log for the session, where a JavaScript error or a failed request on the page often explains a "random" failure.

Using the history#

A test's detail page shows a History strip of its last 20 runs, and All runs opens every run of the test with:

  • a verdict sentence such as "Flaky on Chrome 156: ..." naming where it flakes,
  • the pass rate, number of runs and flips (how often the result changed between consecutive runs on the same environment),
  • a breakdown By environment, which tells you whether the test flakes everywhere or only on one browser,
  • Why it failed, which groups the failures by error message and the step they failed at.

These answer the questions that decide the fix. Flaky only on Safari points at a browser-specific timing or rendering difference. Flaky everywhere since one build points at a change in that build. Failures that all share one error at one step point at one cause; several different errors point at an environment problem or several causes. The Test Analytics guide walks through a full triage, and build analytics show, per build, which failures are new and which tests were already flaky in the builds before, so a flaky test is not blamed on the change under review.

A test's All runs page shows a 100% pass rate on Chrome and Firefox and 65% on Safari, with every failure grouped under one error at the same step. Where do you start?

One browser and one error at one step is a narrow, debuggable signal. Comparing a pass with a fail on that browser shows where the runs diverge; dropping the browser or retrying would discard exactly the evidence that locates the bug.

Infrastructure failures look different#

Not every failure on a grid is the test's fault. Infrastructure problems tend to have a different shape:

  • The session never starts, or fails at "Start Session", rather than at an assertion.
  • The browser disconnects in the middle of an action, with a connection error instead of an assertion or a timeout on a locator.
  • Different tests fail on different runs, each once, rather than the same test repeatedly.

The first two are worth a narrow retry of the session start (chapter 7). If one test fails at the same step with the same error again and again, it is not the infrastructure.

AI Insights, if your account owner has enabled it, adds an explanation of a failed run with a "likely owner" (application, test script, environment or flaky) and a confidence score. The AI Insights documentation is explicit that it is less certain on ambiguous cases such as flaky timing, so treat it as a second opinion next to the comparison above, not as a verdict.

Your tests report a result to TestingBot, but the All runs page never labels anything flaky, even for a test you know fails half the time. What is the most likely reason?

History groups runs by exact name. If each run has its own name, each "test" has a single run and nothing can be flaky. Keep names stable and put run-specific values in the build.

Checklist#

  • Every grid session has a stable name, a build and a reported result.
  • When a test flakes, open one passing and one failing run and find the first step where they differ.
  • Read the value at that step: empty or missing means timing, wrong but plausible means data or the app.
  • Use the environment breakdown and grouped errors before touching the code.
  • Retry only failures that happened before your test did anything.