Choosing a Browser Matrix from Real Analytics
On this page
A browser matrix is the list of browser, version, operating system and device combinations your tests must pass on. Too short and you ship bugs to real users; too long and the suite takes an hour, costs real money and nobody waits for it. This chapter turns your own traffic data into a matrix you can defend, and splits it into tiers so the fast part runs on every commit.
Start from your users, not from a global chart#
Global browser market share is the wrong input. A developer tool might see 20% Firefox; a consumer shopping site might see over half its traffic on iPhones; an internal enterprise app might be almost entirely Edge on Windows. Your analytics tool already knows which is yours.
Export a report with four columns for the last 90 days: browser, operating system, device category and users (or sessions). In Google Analytics 4 that is an exploration with the dimensions Browser, Operating system and Device category and the metric Total users. Save it as CSV:
browser,os,device,users
Chrome,Windows,desktop,41210
Safari,iOS,mobile,30544
Chrome,Android,mobile,22810
Chrome,Macintosh,desktop,12406
Safari,Macintosh,desktop,11873
Edge,Windows,desktop,10232
Firefox,Windows,desktop,4120
Samsung Internet,Android,mobile,3315
Safari,iOS,tablet,2964
Chrome,iOS,mobile,2410
Firefox,Macintosh,desktop,1301
Chrome,Linux,desktop,1180
Opera,Windows,desktop,702
Firefox,Linux,desktop,488
(These numbers are an example. Use your own.)
Find the combinations that cover your traffic#
This short script sorts the rows by users, adds up the share, and stops when it reaches a coverage target. It also labels each row with its rendering engine, because the engine is what decides which bugs a configuration can catch:
import csv
import sys
ENGINES = {"Chrome": "Blink", "Edge": "Blink", "Opera": "Blink",
"Samsung Internet": "Blink", "Firefox": "Gecko", "Safari": "WebKit"}
rows = list(csv.DictReader(open(sys.argv[1])))
target = float(sys.argv[2]) if len(sys.argv) > 2 else 0.95
total = sum(int(r["users"]) for r in rows)
covered = 0
for r in sorted(rows, key=lambda r: int(r["users"]), reverse=True):
share = int(r["users"]) / total
covered += share
# Every browser on iOS runs WebKit (outside the EU), whatever its name.
engine = "WebKit" if r["os"] == "iOS" else ENGINES.get(r["browser"], "?")
print(f"{r['browser']:<17} {r['os']:<10} {r['device']:<8} {engine:<7}"
f" {share:6.1%} cumulative {covered:6.1%}")
if covered >= target:
break
$ python3 coverage.py analytics.csv 0.95
Chrome Windows desktop Blink 28.3% cumulative 28.3%
Safari iOS mobile WebKit 21.0% cumulative 49.3%
Chrome Android mobile Blink 15.7% cumulative 65.0%
Chrome Macintosh desktop Blink 8.5% cumulative 73.5%
Safari Macintosh desktop WebKit 8.2% cumulative 81.6%
Edge Windows desktop Blink 7.0% cumulative 88.7%
Firefox Windows desktop Gecko 2.8% cumulative 91.5%
Samsung Internet Android mobile Blink 2.3% cumulative 93.8%
Safari iOS tablet WebKit 2.0% cumulative 95.8%
Nine combinations cover 95% of this site's users. Two observations are typical:
- Mobile Safari is the second-biggest row. On many sites it is the first. A suite that only runs desktop browsers ignores a fifth of these users.
- Firefox is small but it is the only Gecko row. At 2.8% it would be easy to cut. Keep at least one Firefox configuration anyway: it is the only one that can catch a Gecko bug, and an engine bug affects every Firefox user at once.
Your analytics show Safari on macOS at 1% of users. Before you drop it from the matrix, what should you check?
Analytics only count visitors who could load the page and did not block the tracker. A broken checkout in Safari drives Safari users away, which lowers the share and makes the bug look unimportant. Chrome on macOS is Blink, not WebKit, so it tells you nothing about Safari.
Read the analytics critically#
Analytics data has blind spots that bias the matrix:
- Survivorship. If the site is unusable in a browser, those users leave or never come back, and that browser's share shrinks. A low share can be a symptom of exactly the bug you are about to stop testing for.
- Blocked tracking. Safari's tracking prevention, Firefox's enhanced tracking protection and content blockers stop some analytics scripts from loading. Server-side logs or a privacy-friendly counter give a second opinion.
- Contracts and audiences. A customer contract may name a browser, or a public-sector audience may run a locked-down Edge on Windows. Those go in the matrix regardless of their share.
Choose versions, not only browsers#
Chrome, Edge and Firefox update themselves every few weeks, so almost all their users are on the latest one or two versions. TestingBot, like most grids, lets you ask for latest and latest-1 instead of a fixed number, which keeps the matrix current without edits:
MATRIX = [
{"browser": "chrome", "version": "latest", "platform": "WIN11"},
{"browser": "edge", "version": "latest", "platform": "WIN11"},
{"browser": "firefox", "version": "latest", "platform": "WIN11"},
{"browser": "safari", "version": "latest", "platform": "TAHOE"},
{"browser": "chrome", "version": "latest-1", "platform": "SEQUOIA"},
]
Safari is different. It is updated with the operating system, and many people postpone macOS and iOS upgrades for months. If your analytics show a large group on an older macOS or iOS version, add the Safari version that comes with it as its own row (for example Safari 18 on macOS Sequoia next to Safari 26 on macOS Tahoe). The list of browsers on TestingBot shows which versions exist on which platform.
Firefox has one more variant worth knowing: the Extended Support Release (ESR), which many companies deploy to managed desktops. If your audience is corporate, check the Firefox version numbers in your analytics for an ESR cluster.
Which browser most often needs an older version pinned in the matrix because users lag behind the latest release?
Chrome and Edge update themselves in the background, so latest and latest-1 cover nearly everyone. Safari versions are tied to operating system upgrades that many users postpone, so older Safari versions stay in use much longer.
Split the matrix into tiers#
Running every row on every commit is rarely worth it. A common split:
| Tier | When it runs | What it contains |
|---|---|---|
| Smoke | Every push or pull request | One configuration per engine: Chrome on Windows, Firefox, Safari on macOS, plus your biggest mobile row |
| Full | Nightly and before each release | Every row the coverage script picked, including latest-1 and older Safari versions |
| Exploratory | Before a big UI change ships | A manual pass in the long tail: older devices, small screens, unusual locales |
The smoke tier catches most engine bugs within minutes. The full tier catches version and platform specifics before users do. The exploratory tier is where a person with a live remote browser is faster than writing another automated test.
Revisit the matrix every quarter: rerun the script on fresh data, and check whether a browser version has fallen below your threshold or a new one has appeared.
Summary#
- Build the matrix from your own analytics, labelled by engine, up to a coverage target such as 95%.
- Keep one configuration per engine even when its share is small, and be suspicious of shares that are low for a reason.
- Use
latestandlatest-1for self-updating browsers; pin older Safari versions that your users still run. - Run a per-engine smoke tier on every commit and the full matrix nightly or before release.