Skip to main content

Test Analytics dashboard

The Test Analytics dashboard summarises every automated run in your account over a time range: the pass rate and how it changed, the tests that need attention and why they fail, the environments that fail the most, and your recent builds.

Open it with Analytics in the sidebar of the member dashboard. The line under the title tells you what the page covers, for example Every automated run in your account · 42 tests · 9 environments · 31 builds.

The page reads from top to bottom, from the overall answer to the detail behind it:

  1. The digest: one line that says whether the range is healthy.
  2. The numbers: runs, pass rate, failed runs, flaky tests, tests and duration, each against the range before.
  3. Runs per day: a chart of every day in the range.
  4. Needs attention: the tests that are failing, newly failing, flaky or slower.
  5. By environment and Top errors: where and why runs fail.
  6. Recent builds: the newest builds in the range.

Range and filters

The filter bar at the top decides which runs every part of the page is about. A change applies at once: there is no separate search button.

Custom shows a From and To date in place. The Build filter searches all your builds; tick several and press Apply.
Control Options
Range 24h, 7d, 30d (the default), 90d or Custom. The preset ranges end now. Custom shows a From and To date: pick whole days and press Apply. A custom range can span up to 366 days.
Environment Opens the same browser and device picker as the test dashboard. Select one or more browsers, devices and operating systems. With one selected, the button shows its name (for example Chrome 141 · Windows 11); with more, how many are selected.
Build A searchable list of your builds, newest first. Tick one or more and press Apply; Clear removes the selection. The list shows the first 100 matches: type part of a build name to find older builds.
Tag Only runs with this tag, set with the groups capability.
Started by Only runs started by one member of your team. Shown when your account has team members.

The range in dates is shown on the right of the filter bar, for example Sep 15 – Oct 14, 2026 · UTC. All days are UTC days.

When nothing ran in the selection, the page says No runs in this selection. Your earlier runs are still there: widen the range, clear a filter, or use Reset to the last 30 days.

You can also filter by clicking: a name in the By environment table filters the page to that environment.

The digest

The first card sums up the range in a sentence or two, with a label on the left:

The digest says Needs attention: pass rate 98.4%, 4 tests failing and 3 flaky, Safari 26 is 24% of the runs but 67% of the failures, fix Account › order history first. Below it six numbers with their change
The digest and the numbers below it, each against the 30 days before.
Label When
Needs attention The pass rate is under 95%, the pass rate fell by 2 points or more against the range before, or at least one test is failing.
Healthy None of the above.
No runs No automated run finished in the range.

The sentence can say up to four things:

  • The pass rate over the range and how it moved, for example Pass rate 98.4% over 17,262 runs, down 0.8 pts on the range before.
  • How many tests are failing and flaky, and where most of the flaky ones flake, for example 4 tests are failing and 3 are flaky, 2 of them on Safari 26.
  • An environment that fails far more than it runs: Safari 26 is 24% of the runs but 67% of the failures. This is said when an environment's share of the failed runs is 10 points or more above its share of all runs.
  • The test to fix first, with a link to its All runs page: Fix first: Account › order history, which failed 97 of its 411 runs.

The numbers

Below the digest, six numbers describe the range. Most have a change next to them, measured against the range of the same length just before this one: the 30 days before the last 30 days, for example. A green change is an improvement, a red one is worse, and a grey one is neither good nor bad.

Number What it is Change shown
Runs Finished automated runs in the range. The sparkline shows the runs per day. In percent
Pass rate Passed runs out of the runs that passed or failed. Runs without a result (unknown) are left out. The sparkline shows the pass rate per day. In percentage points (pts)
Failed runs Runs that failed. The sparkline shows the failed runs per day. In runs
Flaky tests Tests whose result keeps changing, out of all tests. See how flaky is decided. A + after the number means it is a lower bound: more tests failed than could be read. None
Tests Different test names that ran in the range. In tests
Duration p50 / p95 The median and 95th percentile duration of the passed runs: half of the passed runs took less than p50, 95% took less than p95. p50, in seconds

Runs per day

A bar for every day in the range, with the runs of that day stacked by their result:

Runs per day over 30 days: green passed bars, a red block of 58 failures on Tuesday Oct 6 where the pass-rate line dips to 91.3%, a grey block of unknown results, and shaded weekends
A day with an outage stands out as a red block and a dip in the pass rate. The hover card gives the day's numbers.
  • Light green: passed runs.
  • Red: failed runs.
  • Grey: runs without a result (unknown).
  • Blue line: the day's pass rate, read on the percentage scale on the right. That scale starts at the lowest pass rate in the range (rounded down to 10%, and at most 90%), so a drop from 99% to 94% is easy to see.
  • Shaded columns: weekends, so a quiet Saturday is not mistaken for a problem.

Hover over a day to see its date, the passed, failed and unknown runs, and the pass rate. Click a day to open that day's page: what changed against the day before, hour by hour, and every run.

Needs attention

The tests worth a look, worst first. Every test is the set of runs with the same name, and it gets one state:

Needs attention lists a failing test, three tests new in build main-2514 and two flaky tests, each with its pass rate, last 20 runs, flips, the browsers it fails on and its latest error
Needs attention: what each test's state is, where it fails and why.
State Meaning
Failing Its last 5 runs failed, an environment failed every one of its runs (3 or more), or its latest build failed every run after an earlier build passed.
Newly failing Failing now, and it ran without failing in the range before. When a build broke it, the label names the build: New in 1.42.0.
Flaky On at least one environment, it failed between 10% and 90% of 5 or more runs, and its result flipped at least twice.
Slower The average duration of its passed runs grew by 30% or more against the range before, with at least 5 passed runs in each range.

These are the same rules as the verdict on a test's All runs page, so a test called flaky here is flaky there too. All thresholds are on Metrics and definitions.

Tabs

All lists every test in the table. The Failing, Newly failing, Flaky and Slower tabs narrow it to one state and show how many tests each has; a tab with no tests is not shown. The first 6 tests show; Show all lists the rest.

The errors fill in a moment after the page shows. The tabs narrow the list without reloading the page.

Columns

Column Shows
Test The test name, how many runs it had, and when it last ran.
State Failing, Newly failing (or New in <build>), Flaky or Slower.
Pass rate The test's own pass rate in the range.
Last 20 A bar per run for its last 20 runs, oldest on the left: green passed, red failed, grey unknown.
Flips How often the result changed between consecutive runs on the same environment. More than 6 is highlighted.
Where The browsers or devices it fails on (up to 3, then +N). For a slower test: where it ran.
Why it fails The error of its latest failed run, and the step it failed at, for example step 7 · click. This is filled in a moment after the page shows, because it is read from the run's steps. For a slower test: its average duration before and now, for example avg 41s → 58s.

Click a test name, or All runs →, to open the All runs page of that test.

To keep the page fast, Needs attention reads the runs of the 20 tests that failed most recently (and of tests that got slower), up to 1,000 runs each. When more tests than that failed, the flaky count shows a + and the digest says at least. Narrow the range or filter by build or environment to see the others.

By environment

A row per browser or device, most runs first. This shows whether failures are spread evenly or come from one place.

By environment shows Safari 26 with 24% of the runs and 67% of the failures in red. Top errors groups a connection refused error across 3 checkout tests and 21 runs, followed by four other errors
By environment and Top errors side by side: Safari 26 fails far more than its share, and one outage failed three checkout tests.
Column Shows
Environment The browser and version with its operating system logo, or for a mobile run the device, for example Safari · iPhone 16 (real device). Hover for the full name.
Runs Its runs, and its share of all runs in percent.
Pass rate Its pass rate in the range.
Share of failures Its part of all failed runs. Shown in bold red when it is 10 points or more above its share of runs: this environment fails more than its fair share.
p50 The median duration of its passed runs.
Trend Its pass rate per day.

Click an environment to filter the whole page to it.

A native app test (Espresso, XCUITest, Maestro or Appium with an app) and a browser test on the same device are listed as two environments, because they test different things.

Top errors

The errors that fail the most runs, across all tests that are failing or flaky. Where Needs attention lists tests, Top errors groups their failures by cause: one broken selector or one unreachable server often fails many tests at once.

A run's error is the one its test detail page shows: the last step in error and its message. Errors are grouped by message, with the parts that change between runs taken out, so these count as one error:

  • the (Session info: chrome=141.0…) that Chrome adds,
  • element and session ids (hexadecimal ids of 16 characters or more),
  • durations such as 30000ms or 1.5s.

Each error, most runs first, shows:

  • the error message, and how many runs failed with it,
  • how many tests it failed, and on which browsers or devices (up to 4, then +N more),
  • the first and last day it happened in the range,
  • a bar per day with its failures, so you see whether it started on one day or comes and goes,
  • Runs →: the failed runs of the test with the most of these failures, on its All runs page.

Top errors loads a moment after the rest of the page, because each error is read from the run's steps. It reads the newest 10 failures of up to 30 tests (those with the most failures), and says so under the list when that is a sample.

A failure without a step in error is not grouped. This happens when the test marked itself as failed (a visual or accessibility check, for example) or failed further back than the steps that are read.

Recent builds

The 5 newest builds that ran in the range, newest first:

Recent builds lists main-2514 with 11 failed runs and +4 new failures, then a nightly and three earlier main builds with their pass rate and wall time
Recent builds: main-2514 broke tests that passed in the build before it.
Column Shows
Build The build name and when its first run started.
Runs Its runs in the range.
Pass rate Its pass rate.
Failed Its failed runs.
New failures Tests that failed in this build and passed in the build before it, for example +3. A dash means there is no earlier build in the range to compare with. A flaky test that happened to fail is counted here too; the build page tells flaky tests apart.
Wall time From the start of its first run to the end of its last.

Click a build to open its build page. When no run in the range has a build, the card explains how to set the build capability.

Copy link and CSV export

The range, the filters and the tabs are all part of the page's URL. Copy link copies it, so a team member opens exactly the same view. You can bookmark it too.

Export CSV downloads the runs per day of the current range and filters, as test-analytics-<from>-<to>.csv:

Column Contains
date The UTC day, as YYYY-MM-DD
runs Finished runs that day
passed Passed runs
failed Failed runs
unknown Runs without a result
pass_rate Passed out of passed and failed, in percent (empty when none passed or failed)

For every single run of a day, use the export on the day's page. For the same numbers in your own tools, use the Insights API.

The numbers are kept for 5 minutes per range and filter, so a reload is fast. A run that finishes in the meantime shows up within 5 minutes.

Was this page helpful?
Last updated