---
title: Test Analytics metrics - how flaky, failing and broken tests are decided
description: 'Definitions of every metric in TestingBot Test Analytics: pass rate,
  flaky, failing, broken, newly failing, fixed, slower, flips, p50 and p95 durations,
  error grouping, and the limits of each view.'
source_url:
  html: https://testingbot.com/support/analytics/metrics
  md: https://testingbot.com/support/analytics/metrics.md
---

# Metrics and definitions

Test Analytics labels tests as flaky, failing or broken with fixed rules, not a model, so the same runs always get the same label, and you can check every label against the runs. This page lists those rules and the limits of each view.

## Runs, tests and results

| Term | Meaning |
| --- | --- |
| **Run** (or session) | One finished automated test session: Selenium, Playwright, Puppeteer, Cypress, Appium, Espresso, XCUITest or Maestro. Manual Live sessions are not counted. |
| **Test** | All runs with exactly the same [name](https://testingbot.com/support/web-automate/selenium/test-options#name). `Login` and `Login (retry)` are two tests. |
| **Passed / Failed** | The result your test reported, through a capability, a framework integration or the [REST API](https://testingbot.com/support/api/tests#updatetest). |
| **Unknown** | A run that ended without a result. It counts as a run, but not as a pass or a failure. |
| **Decided run** | A run that passed or failed. |

## Pass rate

**Passed runs ÷ (passed + failed runs)**, in percent, rounded to one decimal. Unknown runs are left out, so a suite that does not report results does not look like it fails. With no decided runs, the pass rate is shown as `—`.

A change in pass rate is shown in **percentage points** (`pts`): from 95% to 92% is `-3 pts`.

## Ranges and comparisons

- All dates and days are **UTC**. A run belongs to the day it started on.
- The preset ranges (`24h`, `7d`, `30d`, `90d`) end now. A custom range is whole UTC days, up to 366 days.
- Every change (`+12%`, `-3 pts`, `+4s`) compares the range with the **range of the same length just before it**. On the [day page](https://testingbot.com/support/analytics/day), that is the whole day before.
- Green means better, red means worse, grey means neither: more runs is not better or worse in itself.

## Durations, p50 and p95

- A run's **duration** is its wall time: from the moment it started to the moment it finished.
- Duration numbers use **passed runs only**. A failed run stops early or hangs until a timeout; either would distort what a normal duration is.
- **p50** is the median: half of the passed runs took less. With an even number of runs, it is the average of the two middle ones.
- **p95** : 95% of the passed runs took less. A passed run above p95 is listed under _Slower than p95_.
- A **build's wall time** is from the start of its first session to the end of its last.

## Flips

A flip is a change of result between two consecutive decided runs of a test **on the same environment** : passed then failed, or failed then passed.

`pass, pass, fail, pass, fail` on Chrome is 3 flips. A test that fails every time has no flips, and neither does one that always passes. Many flips is the signature of a flaky test. More than 6 flips is highlighted in the tables.

## Test states

A test's state is decided from its runs in the range (the newest 1,000 at most), in this order: the first rule that matches wins.

| State | Rule |
| --- | --- |
| **Broken** | The test's **latest build** failed every one of its runs of this test (at least 2, and none unknown), and an earlier build passed. One failed run is not yet a broken build. A build that broke the test and a later build that fixed it is mentioned in the verdict, but the test is then judged on its other runs. |
| **Failing** | The **last 5 runs** all failed (an unknown result ends the streak), **or** an environment failed every one of its runs (at least 3): a browser or device the test does not work on. |
| **Flaky** | On at least one environment with **5 or more decided runs** , **between 10% and 90%** of them failed, and the result **flipped at least twice**. Runs of a broken build are left out. Fewer runs say too little: pass, fail, pass is one failure, not flakiness. |
| **Stable** | None of the above. Failures that do not meet these rules are one-offs. |
| **Unknown** | Every run ended without a result. |

On the [dashboard](https://testingbot.com/support/analytics/guide#attention), a broken test is listed as _Failing_, and two more states are added:

| State | Rule |
| --- | --- |
| **Newly failing** | Failing (or broken) now, and in the range before it had runs but no failure. Labelled _New in \<build\>_ when a build broke it. |
| **Slower** | The average duration of its passed runs grew by **30% or more** against the range before, with at least 5 passed runs in each range. |

Needs attention orders tests worst first: failing, newly failing, flaky (most failures first), slower.

## Changes between days and builds

| Term | Where | Meaning |
| --- | --- | --- |
| **Started failing** / **Newly failing** | [Day page](https://testingbot.com/support/analytics/day) | Failed on this day, and the day before it passed at least once and did not fail. Looser than the dashboard's _Newly failing_: one failure is enough, the test does not have to be failing by the rules above. |
| **Fixed** | Day page | Failed the day before, and passed every run on this day. |
| **New failures** | [Recent builds](https://testingbot.com/support/analytics/guide#builds) | Tests that failed in a build and passed in the build before it (in the order the builds first ran). |
| **Started failing** | [Build page](https://testingbot.com/support/analytics/builds#since) | Failed in this build, and passed on the same environment in the build before. |
| **Flaky** | Build page | Failed in this build, and the test is flaky over its runs in the 30 builds before. |
| **Fixed** | Build page | Failed in the build before, and passes on every environment in this build. |
| **Got slower** | Build page | A session 30% or more over the test's median duration on the same environment in the builds before. |
| **Changed result** | [All runs](https://testingbot.com/support/analytics/insights#runs) | A run whose result differs from the run before it on the same environment. |

## Environments

- An **environment** is a browser and version on an operating system, or a mobile device. Mobile runs are named by their device, with _real device_, _simulator_ or _emulator_ on hover.
- A native app run (Espresso, XCUITest, Maestro, or Appium with an app) and a browser run on the same device are **two environments**.
- **Share of failures** is an environment's part of all failed runs. When it is **10 points or more** above its share of all runs, the environment fails more than its fair share: it is highlighted and named in the digest.

## Error grouping

A failed run's error is the one its test detail page shows: the **last step in error** and its message. Errors are grouped by message, after taking out what changes between runs:

- the `(Session info: chrome=…)` Chrome adds to its messages,
- hexadecimal ids of 16 characters or more, such as element and session ids,
- durations such as `30000ms` or `1.5s`.

A run that the test itself marked as failed (after a visual or accessibility check, for example) has no step in error, so it is counted but not grouped. Errors are read from the run's steps, which is why they appear a moment after the page shows.

## Limits

To keep every page fast on accounts with millions of runs, some parts read a sample. The page always says so when it does.

| Part | Limit |
| --- | --- |
| Custom range | 366 days |
| All runs of a test | The newest 1,000 runs in the range |
| Needs attention | Up to 20 candidate tests: those with the most recent failures, followed by slower-only tests; up to 1,000 runs per test. Counts become lower bounds (`+`, _at least_) when more failed tests exist |
| Top errors | The newest 10 failures of the 30 tests with the most failures; 6 errors shown |
| Why it failed (All runs) | The newest 40 failures |
| Duration percentiles (dashboard) | The newest 50,000 passed runs |
| Recent builds | The 5 newest builds in the range |
| What broke (day page) | 200 tests, those with the most failures |
| The day, hour by hour | The newest 2,000 runs; 5 environment rows plus _Other_ |
| Build page | 5,000 sessions, failures first; the timeline draws the first 1,500 |
| Build history (flaky, slower) | The 30 builds before, for the 40 tests with the most failures and the slowest |
| Freshness | Numbers are kept for 5 minutes per range and filter |

## Looking for more help?

Have questions or need more information? Reach out via email or Slack.

[Email us](https://testingbot.com/contact/new) [Join our Slack](https://join.slack.com/t/testingb0t/shared_invite/zt-3bcw9xch-jk19~6XPs_xBrsAgAedkCw)