---
title: Run the Matrix in Parallel on TestingBot | TestingBot
description: 'A cross-browser suite multiplies: every test runs once per configuration.
  Twenty tests on five configurations are a hundred browser sessions, and at half
  a…'
source_url:
  html: https://testingbot.com/resources/courses/cross-browser-testing/run-the-matrix-in-parallel
  md: https://testingbot.com/resources/courses/cross-browser-testing/run-the-matrix-in-parallel.md
---

Course

[Cross-Browser Testing Tutorial](https://testingbot.com/resources/courses/cross-browser-testing)

Chapter 8 of 11

1. [1 Cross-Browser Testing Tutorial](https://testingbot.com/resources/courses/cross-browser-testing)
2. [2 Chromium, Gecko and WebKit: What Really Differs](https://testingbot.com/resources/courses/cross-browser-testing/browser-engines)
3. [3 Choosing a Browser Matrix from Real Analytics](https://testingbot.com/resources/courses/cross-browser-testing/choosing-a-browser-matrix)
4. [4 Writing Tests That Survive Engine Differences](https://testingbot.com/resources/courses/cross-browser-testing/tests-that-survive-engine-differences)
5. [5 Selenium Across Browsers and Versions](https://testingbot.com/resources/courses/cross-browser-testing/selenium-across-browsers)
6. [6 Playwright Projects and Branded Browsers](https://testingbot.com/resources/courses/cross-browser-testing/playwright-across-browsers)
7. [7 Mobile Browsers: Safari on iOS, Chrome on Android](https://testingbot.com/resources/courses/cross-browser-testing/mobile-browsers)
8. [8 Run the Matrix in Parallel on TestingBot](https://testingbot.com/resources/courses/cross-browser-testing/run-the-matrix-in-parallel)
9. [9 Debugging a Browser-Specific Failure](https://testingbot.com/resources/courses/cross-browser-testing/debug-a-browser-specific-failure)
10. [10 Visual Differences Between Browsers](https://testingbot.com/resources/courses/cross-browser-testing/visual-differences)
11. [11 Knowledge Check](https://testingbot.com/resources/courses/cross-browser-testing/knowledge-check)

[Exit course](https://testingbot.com/resources/courses)

 Chapter 8 of 11 Intermediate 6 min read 2 quick checks Updated Nov 2026 

# Run the Matrix in Parallel on TestingBot
On this page
1. [The arithmetic](https://testingbot.com#the-arithmetic)
2. [Tiers in the matrix](https://testingbot.com#tiers-in-the-matrix)
3. [Parallel Selenium with pytest-xdist](https://testingbot.com#parallel-selenium-with-pytest-xdist)
4. [Parallel Playwright](https://testingbot.com#parallel-playwright)
5. [Data the parallel run gives you](https://testingbot.com#data-the-parallel-run-gives-you)
6. [Summary](https://testingbot.com#summary)

A cross-browser suite multiplies: every test runs once per configuration. Twenty tests on five configurations are a hundred browser sessions, and at half a minute each that is fifty minutes one after another. Nobody waits fifty minutes for a pull request. This chapter makes the matrix fast enough to run on every push, by running sessions in parallel and by running only the tier that the moment needs.

## The arithmetic[#](https://testingbot.com#the-arithmetic)

Three numbers decide how long a run takes:

- **Sessions** : tests multiplied by configurations.
- **Average session length** , including browser start-up.
- **Parallel slots** : how many sessions run at the same time.

Wall-clock time is roughly sessions times average length, divided by slots. In chapter 4, the login test on five configurations took 47.7 seconds with one session at a time, about 9.5 seconds per session. With five slots it would take about as long as the slowest single session.

Parallel slots are what your TestingBot plan provides. If a run asks for more sessions than you have slots, TestingBot queues the extra ones (for up to six minutes) and starts them as slots free up, so asking for too many is slower, not broken. Queue time is not billed.

Your plan has 4 parallel slots. You run 40 tests on 5 configurations with `-n 10`. Roughly what happens?

 Four sessions run at a time and the other six workers wait in the queue; the run is no faster than with `-n 4` Six workers fail immediately with a connection error The run is two and a half times faster than with `-n 4`

Sessions beyond your slots are queued, not rejected, and they only start when a slot frees up. More workers than slots adds waiting, not speed. Set the worker count to your slot count.

## Tiers in the matrix[#](https://testingbot.com#tiers-in-the-matrix)

Chapter 2 split the matrix into a smoke tier for every push and a full tier for nightly runs. Put the tier in the data, and let an environment variable choose:

```python
import os

# One entry per browser/OS combination. "smoke" runs on every push, "full" nightly.
MATRIX = [
    {"browser": "chrome", "version": "latest", "platform": "WIN11", "tier": "smoke"},
    {"browser": "firefox", "version": "latest", "platform": "WIN11", "tier": "smoke"},
    {"browser": "safari", "version": "latest", "platform": "TAHOE", "tier": "smoke"},
    {"browser": "edge", "version": "latest", "platform": "WIN11", "tier": "full"},
    {"browser": "chrome", "version": "latest-1", "platform": "SEQUOIA", "tier": "full"},
]

TIER = os.environ.get("TB_TIER", "smoke")
SELECTED = [c for c in MATRIX if TIER == "full" or c["tier"] == "smoke"]
```

In `conftest.py`, import `SELECTED` instead of `MATRIX` and parametrize over it: `@pytest.fixture(params=SELECTED, ids=config_id)`. Check what each tier would run without starting a single browser:

```bash
$ pytest --collect-only -q tests/test_login.py
tests/test_login.py::test_login[chrome-latest-WIN11]
tests/test_login.py::test_login[firefox-latest-WIN11]
tests/test_login.py::test_login[safari-latest-TAHOE]

3 tests collected in 0.10s

$ TB_TIER=full pytest --collect-only -q tests/test_login.py
...
5 tests collected in 0.07s
```

The smoke tier is one configuration per engine. It catches most cross-browser bugs in a fraction of the time, and the full tier picks up the rest overnight.

## Parallel Selenium with pytest-xdist[#](https://testingbot.com#parallel-selenium-with-pytest-xdist)

pytest-xdist runs tests in several worker processes. Each worker creates its own driver through the fixture, so each worker is one remote session at a time:

```bash
pip install pytest-xdist
export TB_KEY=your-key TB_SECRET=your-secret
TB_BUILD="smoke-$(git rev-parse --short HEAD)" pytest -n 4
```

Set `-n` to your plan's parallel slots. Nothing in the tests changes, because the fixture from chapter 4 already does everything per test: it opens its own session, names it after the test and its configuration, and reports the result.

Two habits make parallel runs readable:

- **One build per run.** A build name with the commit hash (or your CI run id) groups every session of the run in the dashboard. The [build option](https://testingbot.com/support/web-automate/selenium/test-options#build) is all it takes.
- **Readable session names.** The parametrized test id, such as `test_login[safari-latest-TAHOE]`, says which configuration failed without opening the session.

For more xdist options, see the [pytest guide for TestingBot](https://testingbot.com/support/web-automate/selenium/python/pytest) and the [parallel testing chapter of the Selenium Python course](https://testingbot.com/resources/courses/selenium-python/parallel-with-pytest-xdist).

Tests pass one at a time but fail randomly with `-n 4`: two tests that edit the same user profile overwrite each other. What is the right fix?

 Run that file with `-n 1` forever Give each test its own data, for example a user created per test or per worker Add retries until it passes

Separate browser sessions isolate cookies and storage, not your backend. With the matrix, the same test also runs on several browsers at once, so shared accounts collide even within one test. Unique data per test is the only lasting fix.

## Parallel Playwright[#](https://testingbot.com#parallel-playwright)

In Playwright, the `workers` setting does the same job. On a remote grid each worker opens one session per project and reuses it for the tests it runs, so the number of concurrent sessions is the number of workers:

```bash
TB_BUILD="smoke-$(git rev-parse --short HEAD)" \
  npx playwright test -c playwright.testingbot.config.ts --workers=4
```

`fullyParallel: true` lets Playwright spread the tests of one file over several workers, which keeps all slots busy when one file has most of the tests. Each test still gets a fresh browser context, so tests do not share cookies or storage even when they share a session.

## Data the parallel run gives you[#](https://testingbot.com#data-the-parallel-run-gives-you)

A parallel run with a build name gives you a table you can read at a glance: tests down the side, configurations across the top. Three patterns are worth recognising:

| Pattern | Usually means |
| --- | --- |
| One test fails in every configuration | A real bug or a broken test, not a cross-browser issue |
| Many tests fail in one configuration | A browser-specific bug, or a test habit from chapter 3 that this browser exposes |
| Different tests fail on each run, in any configuration | Flakiness: shared data, missing waits or a slow environment |

Only the middle row is a cross-browser problem. The next chapter takes one of those and debugs it.

## Summary[#](https://testingbot.com#summary)

- Sessions are tests times configurations; wall time is sessions times length divided by parallel slots.
- Sessions beyond your slots are queued, so set workers to your slot count.
- Keep tiers in the matrix data and select them with an environment variable; check the selection with `--collect-only`.
- Use pytest-xdist (`-n`) or Playwright `workers`, one build name per run, and unique test data.

[Previous · Chapter 7 Mobile Browsers: Safari on iOS, Chrome on Android](https://testingbot.com/resources/courses/cross-browser-testing/mobile-browsers) [Next · Chapter 9 Debugging a Browser-Specific Failure](https://testingbot.com/resources/courses/cross-browser-testing/debug-a-browser-specific-failure)

On this page

1. [The arithmetic](https://testingbot.com#the-arithmetic)
2. [Tiers in the matrix](https://testingbot.com#tiers-in-the-matrix)
3. [Parallel Selenium with pytest-xdist](https://testingbot.com#parallel-selenium-with-pytest-xdist)
4. [Parallel Playwright](https://testingbot.com#parallel-playwright)
5. [Data the parallel run gives you](https://testingbot.com#data-the-parallel-run-gives-you)
6. [Summary](https://testingbot.com#summary)

Certification

Earn the free, resume-worthy TestingBot Cross-Browser Testing Certification for this course.

[How it works](https://testingbot.com/resources/courses/cross-browser-testing/certification)
