Skip to main content

All runs of a test

The All runs page shows every run of one test, so you can decide in seconds whether it is stable, flaky, failing or broken by a build, on which browsers and devices, since when, and with which error. It replaces the scatter plot of the former Test Insights page.

All runs of Search › autocomplete suggestions: the Flaky verdict, five numbers against the previous 30 days, and a chart of the duration of every run with failed runs as red diamonds
All runs of a flaky test: the verdict names the browser and the error, the chart shows every run.

A test is every run with exactly the same name. Name your tests with the name capability, or through the API, and use the same name every time the test runs. Tests without a name (unnamed test…) have no history to show.

There are several ways to open it:

  • On a test's detail page, the History strip in the header shows its last 20 runs as bars, oldest left. Click All runs next to it.
  • On the Test Analytics dashboard, click a test in Needs attention, or Runs → under an error in Top errors.
  • On a day page, click All runs next to a run or a fixed test.
  • On a build page, click Runs → next to a test under Since <build>, or All runs → next to a session.

Range and filters

Control Options
Range 24h, 7d, 30d (the default), 90d or Custom: a From and To date, up to 366 days.
Environment Only the runs on one browser or device. The list holds the environments this test ran on in the range.
Build Only the runs of one build. The list holds the builds this test ran in in the range.
Result Passed or Failed runs only.

A range holds up to the newest 1,000 runs of the test. When it has more, the page says so under the verdict: pick a shorter range to see all of them.

The verdict

The card at the top reads the runs for you and sums them up in one sentence, with a label. You do not have to study the chart to triage a test.

Flaky on Safari 26: 34 of 97 runs failed, all on element click intercepted at step 5. Below: pass rate 91.7%, 411 runs, 39 flips on Safari, duration and the last failure 47 minutes ago
A flaky test: where it flakes, how often, and the error it flakes with.
Broken since build main-2514 on Chrome 148, FF 146, and Safari 26: all 3 of its runs failed
A test broken by a build: the build and the browsers it fails on.
Label When Example
Broken The latest build failed every one of its runs (at least 2), and an earlier build passed. Broken since build 1.42.0 on every browser: all 6 of its runs failed.
Failing The last 5 runs or more all failed, or an environment failed every one of its runs (at least 3). Failing on Safari 18: all 7 of its runs failed, since Oct 2.
Flaky On an environment with 5 or more passed or failed runs, between 10% and 90% failed, and the result changed at least twice. Flaky on Firefox 143: 4 of 19 runs failed.
Stable None of the above. Occasional failures are called one-offs. Stable: 58 of 59 runs passed; the failure was a one-off.
Unknown Every run ended without a result. No result: all 12 runs ended without passing or failing.
No runs The test did not finish a run in this range.

Once the errors are read (a moment after the page shows), the sentence also names the error when all failures share one, for example …, all on element not interactable at step 11. When an earlier build broke the test and a later build fixed it, the verdict says that too: Separately, build 1.40.2 broke it on Chrome 141: all 3 of its runs failed, until 1.40.3 fixed it.

Show the failures opens the Failed tab of Runs. All rules and thresholds are on Metrics and definitions.

The numbers

Each number is compared with the range of the same length just before this one.

Number What it is
Pass rate Passed out of passed and failed runs, with the change in points.
Runs The test's runs in the range, and how many failed or have no result.
Flips How often the result changed between consecutive runs on the same environment, for the environment where it changed the most (named next to the number). A stable or a constantly failing test has few flips; a flaky test has many.
Duration p50 / p95 The median and 95th percentile duration of the passed runs, with the change of the median in seconds.
Last failure How long ago the test last failed, and where (environment and build). Click it to open that run.

Duration per run

A chart with a mark for every run, placed at the time it started (across) and its duration (up):

The verdict names the error once the failures are read. Hover over a run to see where and why it failed.
  • Green dot: passed. Red diamond: failed. Grey dot: no result.
  • Blue line: the median duration of the passed runs, moving along the range, so you see when the test got slower.
  • Blue band: from p50 to p95 of the passed runs. A passed run above the band is slower than 95% of the others. With too few passed runs for a moving median, a dashed line shows the median of all passed runs instead.
  • Dashed vertical lines: where each build first ran, labelled with the build name.
  • Shaded area: a build that failed every run, up to the build that fixed it (or to the end of the range if it is still broken).
  • A run far slower than all others, such as one that hung until a timeout, is drawn on the top edge so it does not flatten the rest of the chart.

The legend only lists what the chart draws. Hover over a mark to see its result and environment, when it ran, its build and duration, and for a failed run its error. Click a mark to open the run in the test panel next to the page.

Duration chart of Checkout › pays with saved card: a cluster of fast failures on Oct 6 and, at the right edge, the window of build main-2514 shaded red and labelled broken by main-2514
A broken test: the window of the build that broke it is shaded, up to now.

By environment and per day

By environment has a row per browser or device the test ran on:

By environment shows Safari 26 at 64.9% with red bars among its last 20 runs and 39 flips, the other six environments at 100%; Why it failed groups 34 runs under element click intercepted at step 5
The flaky test only fails on Safari 26, always with the same error.
  • Pass rate on that environment.
  • Last 20: its last 20 runs there, oldest on the left: green passed, red failed, grey unknown. A flaky test shows red bars scattered between green ones; a failing one shows red at the end.
  • p50: the median duration of its passed runs there.
  • Flips: how often its result changed there. More than 6 is highlighted.

Per day shows the test's passed (green), failed (red) and unknown (grey) runs for each day of the range. Hover over a day for its numbers.

Why it failed

The test's failures, grouped by their error, most runs first. The error of a run is the one its detail page shows: the last step in error and its message. Parts of a message that change from run to run (the Chrome version Chrome adds, element and session ids, durations) are ignored, so the same failure is one group.

Each group shows:

  • the error message and how many runs failed with it,
  • the browsers or devices it happened on,
  • new in <build> when it first happened in a build after the start of the range: the build that introduced it,
  • the step it failed at, for example step 11,
  • the first and last day it happened, and a bar per day with its failures,
  • Latest run →, which opens its latest run in the test panel.

The errors are read from the steps of each run, so this card fills a moment after the page shows. It groups the newest 40 failures; the Failed tab of Runs lists all of them. A failure without a step in error (the test marked itself as failed, for example after a visual or accessibility check) is counted under the list, but not grouped.

Runs

Every run of the test in the range, newest first:

The Runs table with All, Failed, Slower than p95 and Changed result tabs; the run you came from is highlighted; a failed Safari 26 run shows step 5 · click
Runs, newest first. The run you came from is highlighted.
Column Shows
Started When the run started (UTC) and how long ago. The run you came from is highlighted and marked the run you came from.
Result Passed, Failed or Unknown.
Environment The browser or device, and its operating system.
Build The build it ran in, if any.
Duration The duration, with a bar: red for a failed run, amber for a passed run slower than p95, grey otherwise. The tick on the bar marks the median.
Failed at For a failed run, the step and command it failed at, for example step 7 · click.

The tabs narrow the list and show how many runs each has:

Tab Lists
All Every run.
Failed The failed runs.
Slower than p95 Passed runs that took longer than the p95 duration.
Changed result Runs whose result differs from the run before them on the same environment: the moments it started failing or passing again.

The first 12 runs show; Load 25 more adds the next ones. Click a run to open it in the test panel next to the list, with its video, steps and logs. Back to the run at the top returns to the run you came from.

Copy link and CSV export

The range, filters and the open tab are part of the URL: Copy link copies it to share this exact view.

Export CSV downloads the runs in the current range, filters and tab, newest first, named after the test:

Column Contains
started_at When the run started, in UTC (ISO 8601)
result passed, failed or unknown
browser The browser name and version
os The operating system or device
build The build name, if any
duration_seconds From start to finish, in seconds
session_id The session id, as used by the REST API
url The link to the run in the member dashboard
Was this page helpful?
Last updated