Run the Matrix in Parallel on TestingBot
On this page
A cross-browser suite multiplies: every test runs once per configuration. Twenty tests on five configurations are a hundred browser sessions, and at half a minute each that is fifty minutes one after another. Nobody waits fifty minutes for a pull request. This chapter makes the matrix fast enough to run on every push, by running sessions in parallel and by running only the tier that the moment needs.
The arithmetic#
Three numbers decide how long a run takes:
- Sessions: tests multiplied by configurations.
- Average session length, including browser start-up.
- Parallel slots: how many sessions run at the same time.
Wall-clock time is roughly sessions times average length, divided by slots. In chapter 4, the login test on five configurations took 47.7 seconds with one session at a time, about 9.5 seconds per session. With five slots it would take about as long as the slowest single session.
Parallel slots are what your TestingBot plan provides. If a run asks for more sessions than you have slots, TestingBot queues the extra ones (for up to six minutes) and starts them as slots free up, so asking for too many is slower, not broken. Queue time is not billed.
Your plan has 4 parallel slots. You run 40 tests on 5 configurations with -n 10. Roughly what happens?
Sessions beyond your slots are queued, not rejected, and they only start when a slot frees up. More workers than slots adds waiting, not speed. Set the worker count to your slot count.
Tiers in the matrix#
Chapter 2 split the matrix into a smoke tier for every push and a full tier for nightly runs. Put the tier in the data, and let an environment variable choose:
import os
# One entry per browser/OS combination. "smoke" runs on every push, "full" nightly.
MATRIX = [
{"browser": "chrome", "version": "latest", "platform": "WIN11", "tier": "smoke"},
{"browser": "firefox", "version": "latest", "platform": "WIN11", "tier": "smoke"},
{"browser": "safari", "version": "latest", "platform": "TAHOE", "tier": "smoke"},
{"browser": "edge", "version": "latest", "platform": "WIN11", "tier": "full"},
{"browser": "chrome", "version": "latest-1", "platform": "SEQUOIA", "tier": "full"},
]
TIER = os.environ.get("TB_TIER", "smoke")
SELECTED = [c for c in MATRIX if TIER == "full" or c["tier"] == "smoke"]
In conftest.py, import SELECTED instead of MATRIX and parametrize over it: @pytest.fixture(params=SELECTED, ids=config_id). Check what each tier would run without starting a single browser:
$ pytest --collect-only -q tests/test_login.py
tests/test_login.py::test_login[chrome-latest-WIN11]
tests/test_login.py::test_login[firefox-latest-WIN11]
tests/test_login.py::test_login[safari-latest-TAHOE]
3 tests collected in 0.10s
$ TB_TIER=full pytest --collect-only -q tests/test_login.py
...
5 tests collected in 0.07s
The smoke tier is one configuration per engine. It catches most cross-browser bugs in a fraction of the time, and the full tier picks up the rest overnight.
Parallel Selenium with pytest-xdist#
pytest-xdist runs tests in several worker processes. Each worker creates its own driver through the fixture, so each worker is one remote session at a time:
pip install pytest-xdist
export TB_KEY=your-key TB_SECRET=your-secret
TB_BUILD="smoke-$(git rev-parse --short HEAD)" pytest -n 4
Set -n to your plan's parallel slots. Nothing in the tests changes, because the fixture from chapter 4 already does everything per test: it opens its own session, names it after the test and its configuration, and reports the result.
Two habits make parallel runs readable:
- One build per run. A build name with the commit hash (or your CI run id) groups every session of the run in the dashboard. The build option is all it takes.
-
Readable session names. The parametrized test id, such as
test_login[safari-latest-TAHOE], says which configuration failed without opening the session.
For more xdist options, see the pytest guide for TestingBot and the parallel testing chapter of the Selenium Python course.
Tests pass one at a time but fail randomly with -n 4: two tests that edit the same user profile overwrite each other. What is the right fix?
Separate browser sessions isolate cookies and storage, not your backend. With the matrix, the same test also runs on several browsers at once, so shared accounts collide even within one test. Unique data per test is the only lasting fix.
Parallel Playwright#
In Playwright, the workers setting does the same job. On a remote grid each worker opens one session per project and reuses it for the tests it runs, so the number of concurrent sessions is the number of workers:
TB_BUILD="smoke-$(git rev-parse --short HEAD)" \
npx playwright test -c playwright.testingbot.config.ts --workers=4
fullyParallel: true lets Playwright spread the tests of one file over several workers, which keeps all slots busy when one file has most of the tests. Each test still gets a fresh browser context, so tests do not share cookies or storage even when they share a session.
Data the parallel run gives you#
A parallel run with a build name gives you a table you can read at a glance: tests down the side, configurations across the top. Three patterns are worth recognising:
| Pattern | Usually means |
|---|---|
| One test fails in every configuration | A real bug or a broken test, not a cross-browser issue |
| Many tests fail in one configuration | A browser-specific bug, or a test habit from chapter 3 that this browser exposes |
| Different tests fail on each run, in any configuration | Flakiness: shared data, missing waits or a slow environment |
Only the middle row is a cross-browser problem. The next chapter takes one of those and debugs it.
Summary#
- Sessions are tests times configurations; wall time is sessions times length divided by parallel slots.
- Sessions beyond your slots are queued, so set workers to your slot count.
- Keep tiers in the matrix data and select them with an environment variable; check the selection with
--collect-only. - Use pytest-xdist (
-n) or Playwrightworkers, one build name per run, and unique test data.