Metrics and definitions

Test Analytics labels tests as flaky, failing or broken with fixed rules, not a model, so the same runs always get the same label, and you can check every label against the runs. This page lists those rules and the limits of each view.

Runs, tests and results

Term	Meaning
Run (or session)	One finished automated test session: Selenium, Playwright, Puppeteer, Cypress, Appium, Espresso, XCUITest or Maestro. Manual Live sessions are not counted.
Test	All runs with exactly the same name. `Login` and `Login (retry)` are two tests.
Passed / Failed	The result your test reported, through a capability, a framework integration or the REST API.
Unknown	A run that ended without a result. It counts as a run, but not as a pass or a failure.
Decided run	A run that passed or failed.

Pass rate

Passed runs ÷ (passed + failed runs), in percent, rounded to one decimal. Unknown runs are left out, so a suite that does not report results does not look like it fails. With no decided runs, the pass rate is shown as —.

A change in pass rate is shown in percentage points (pts): from 95% to 92% is -3 pts.

Ranges and comparisons

All dates and days are UTC. A run belongs to the day it started on.
The preset ranges (24h, 7d, 30d, 90d) end now. A custom range is whole UTC days, up to 366 days.
Every change (+12%, -3 pts, +4s) compares the range with the range of the same length just before it. On the day page, that is the whole day before.
Green means better, red means worse, grey means neither: more runs is not better or worse in itself.

Durations, p50 and p95

A run's duration is its wall time: from the moment it started to the moment it finished.
Duration numbers use passed runs only. A failed run stops early or hangs until a timeout; either would distort what a normal duration is.
p50 is the median: half of the passed runs took less. With an even number of runs, it is the average of the two middle ones.
p95: 95% of the passed runs took less. A passed run above p95 is listed under Slower than p95.
A build's wall time is from the start of its first session to the end of its last.

Flips

A flip is a change of result between two consecutive decided runs of a test on the same environment: passed then failed, or failed then passed.

pass, pass, fail, pass, fail on Chrome is 3 flips. A test that fails every time has no flips, and neither does one that always passes. Many flips is the signature of a flaky test. More than 6 flips is highlighted in the tables.

Test states

A test's state is decided from its runs in the range (the newest 1,000 at most), in this order: the first rule that matches wins.

State	Rule
Broken	The test's latest build failed every one of its runs of this test (at least 2, and none unknown), and an earlier build passed. One failed run is not yet a broken build. A build that broke the test and a later build that fixed it is mentioned in the verdict, but the test is then judged on its other runs.
Failing	The last 5 runs all failed (an unknown result ends the streak), or an environment failed every one of its runs (at least 3): a browser or device the test does not work on.
Flaky	On at least one environment with 5 or more decided runs, between 10% and 90% of them failed, and the result flipped at least twice. Runs of a broken build are left out. Fewer runs say too little: pass, fail, pass is one failure, not flakiness.
Stable	None of the above. Failures that do not meet these rules are one-offs.
Unknown	Every run ended without a result.

On the dashboard, a broken test is listed as Failing, and two more states are added:

State	Rule
Newly failing	Failing (or broken) now, and in the range before it had runs but no failure. Labelled New in <build> when a build broke it.
Slower	The average duration of its passed runs grew by 30% or more against the range before, with at least 5 passed runs in each range.

Needs attention orders tests worst first: failing, newly failing, flaky (most failures first), slower.

Changes between days and builds

Term	Where	Meaning
Started failing / Newly failing	Day page	Failed on this day, and the day before it passed at least once and did not fail. Looser than the dashboard's Newly failing: one failure is enough, the test does not have to be failing by the rules above.
Fixed	Day page	Failed the day before, and passed every run on this day.
New failures	Recent builds	Tests that failed in a build and passed in the build before it (in the order the builds first ran).
Started failing	Build page	Failed in this build, and passed on the same environment in the build before.
Flaky	Build page	Failed in this build, and the test is flaky over its runs in the 30 builds before.
Fixed	Build page	Failed in the build before, and passes on every environment in this build.
Got slower	Build page	A session 30% or more over the test's median duration on the same environment in the builds before.
Changed result	All runs	A run whose result differs from the run before it on the same environment.

Environments

An environment is a browser and version on an operating system, or a mobile device. Mobile runs are named by their device, with real device, simulator or emulator on hover.
A native app run (Espresso, XCUITest, Maestro, or Appium with an app) and a browser run on the same device are two environments.
Share of failures is an environment's part of all failed runs. When it is 10 points or more above its share of all runs, the environment fails more than its fair share: it is highlighted and named in the digest.

Error grouping

A failed run's error is the one its test detail page shows: the last step in error and its message. Errors are grouped by message, after taking out what changes between runs:

the (Session info: chrome=…) Chrome adds to its messages,
hexadecimal ids of 16 characters or more, such as element and session ids,
durations such as 30000ms or 1.5s.

A run that the test itself marked as failed (after a visual or accessibility check, for example) has no step in error, so it is counted but not grouped. Errors are read from the run's steps, which is why they appear a moment after the page shows.

Limits

To keep every page fast on accounts with millions of runs, some parts read a sample. The page always says so when it does.

Part	Limit
Custom range	366 days
All runs of a test	The newest 1,000 runs in the range
Needs attention	Up to 20 candidate tests: those with the most recent failures, followed by slower-only tests; up to 1,000 runs per test. Counts become lower bounds (`+`, at least) when more failed tests exist
Top errors	The newest 10 failures of the 30 tests with the most failures; 6 errors shown
Why it failed (All runs)	The newest 40 failures
Duration percentiles (dashboard)	The newest 50,000 passed runs
Recent builds	The 5 newest builds in the range
What broke (day page)	200 tests, those with the most failures
The day, hour by hour	The newest 2,000 runs; 5 environment rows plus Other
Build page	5,000 sessions, failures first; the timeline draws the first 1,500
Build history (flaky, slower)	The 30 builds before, for the 40 tests with the most failures and the slowest
Freshness	Numbers are kept for 5 minutes per range and filter