Name: Selenium with Python and pytest
Availability: InStock

A suite of 200 browser tests at 20 seconds each needs over an hour on one process. Spread across ten browsers it needs a few minutes. pytest-xdist makes the parallel part a command-line flag. Making the tests safe to run side by side is the real work, and it comes down to one rule: every test gets its own browser, and nothing mutable is shared between tests.

Turning it on#

pytest-xdist is already in requirements.txt. Pass -n with a number of worker processes:

pytest --headless -n 4

or let it pick one worker per CPU core:

pytest --headless -n auto

On our 16-core machine, -n auto started 16 workers and the seven-test suite dropped from 9.2 seconds to 4.2. The gain is smaller than 16x because each test still starts its own browser and the slowest test sets the floor; on a larger suite the ratio improves.

-n 0 turns xdist off again, which is the quickest way to tell whether a failure is a parallelism problem: if a test fails with -n 4 and passes with -n 0, look for shared state.

How xdist runs your tests#

xdist starts separate Python processes, not threads. Each worker imports your test files and conftest.py, collects the same list of tests, and then runs the subset the controller hands it. Consequences:

Module-level variables are per worker. A global counter or list in a test file is not shared between workers, and it is shared between all the tests that happen to land on the same worker. Neither is what you want.
Session-scoped fixtures run once per worker, not once per run. A session fixture that creates a test user creates one per worker. If something must truly happen once, the pytest-xdist docs show a FileLock pattern; usually it is simpler to make the setup safe to run more than once.
Order is not guaranteed. With the default --dist load, any worker may run any test, in any order. A test that only passes after another test is broken already; xdist just exposes it.
Output is quiet. print and -s output from workers is not shown live; it appears in failure reports.

Each worker knows its name through the PYTEST_XDIST_WORKER environment variable (gw0, gw1, ...) and the worker_id fixture. The screenshot helper from chapter 5 already uses it in file names.

A session-scoped fixture signs up a user with a fixed email address. The suite passes with -n 0 and fails with "email already taken" under -n 4. Why?

xdist runs tests in threads, and the fixture is not thread-safe. Each worker process runs session fixtures itself, so four workers try to sign up the same address. Session-scoped fixtures are not supported with xdist.

xdist workers are separate processes, and "session" means each worker's session. Generate unique data per worker or per test (for example with uuid4()), or guard truly global setup with a file lock.

Distribution modes#

--dist controls how tests are handed out:

Mode	Guarantee	Use when
`load` (default)	none; any test, any worker	tests are independent, which they should be
`loadscope`	tests of one module (or class) run on one worker	module- or class-scoped fixtures are expensive
`loadfile`	tests of one file run on one worker	the same, grouped by file
`loadgroup`	tests with the same `@pytest.mark.xdist_group("name")` run on one worker	a few tests must share something; the rest stay spread out
`worksteal`	like `load`, but idle workers take pending tests from busy ones	test durations vary a lot

pytest --headless -n 3 --dist loadfile

Reach for these deliberately. If you find you need loadscope to make the suite pass, that is a sign tests depend on each other; fixing the dependency is usually better than pinning it to a worker.

What must not be shared#

The driver fixture is function-scoped, so each test already gets its own browser in its own worker. These are the less obvious shared resources behind "only fails in parallel" bugs:

Test accounts. Two tests logging in as the same user can log each other out, and two tests editing the same record race each other. Create data per test, through an API in a fixture, with a unique suffix such as f"user-{uuid.uuid4().hex[:8]}".
Files. Screenshots, downloads and reports written to fixed names overwrite each other. Include the node ID and worker in the path.
Server-side state. A test that empties a shopping cart or changes a global setting affects every test running against the same environment at that moment.
Ports. A fixture that starts a local server on port 8000 collides with itself on the second worker. Bind to port 0 and read back the port the OS chose.
Cached page objects. A page object stored at module level wraps one test's driver. Create page objects inside tests, from the driver fixture.

With -n 4, two tests that each edit "the demo project" fail now and then with unexpected values. They pass alone. What is the right fix?

Add time.sleep(1) before each edit. Raise the explicit wait timeouts. Give each test its own project, created in a fixture, so the tests no longer share data.

This is a data race between two tests on the same record, not a timing problem inside one test. Sleeps and longer waits change when the collision happens, not whether it can. Isolated data removes it.

Cross-browser and parallel together#

The --browser option from chapter 4 combines with -n. Every test is parametrised once per browser, and xdist spreads all of the resulting tests over the workers:

pytest --headless --browser chrome --browser firefox -n 4

14 passed in 12.89s

Seven tests times two browsers, four at a time.

How many workers?#

Locally, each worker runs a real browser, which needs CPU and memory. Past a point, extra workers make every test slower, explicit waits start timing out, and the suite becomes flaky instead of fast. -n auto is a reasonable start for headless runs on a developer machine; on a shared CI agent, start lower and measure.
On a remote grid the browsers run elsewhere and your machine only sends commands, so the limit becomes the number of sessions your grid account allows at once. The next chapter covers what happens when you ask for more. TestingBot's parallel calculator estimates how many slots a suite needs for a target run time.

Speeding up the suite with more local browsers only goes so far, and it still only tests the browsers installed on your machine. The last content chapter moves the browsers to a cloud grid.