Parallel Selenium tests with NUnit
On this page
Browser tests are slow because browsers are slow: most of a test's time is spent starting the browser and waiting for pages. The tests themselves barely use the CPU, so running several at once is the cheapest speed-up there is. NUnit can do it with two attributes, but only if every test has its own browser. This chapter shows the bug you get when they don't, and the fix.
NUnit runs tests one at a time by default#
Without any attribute, NUnit runs every test sequentially on one thread. Parallelism is opt-in with [Parallelizable], at one of these scopes:
| Scope | Runs in parallel with others |
|---|---|
ParallelScope.Self |
this fixture (or test) |
ParallelScope.Children |
the tests inside this fixture, with each other |
ParallelScope.Fixtures |
the fixtures inside this assembly or namespace, with each other; tests inside one fixture still run one at a time |
ParallelScope.All |
everything: fixtures and the tests within them |
You can put the attribute on individual fixtures, or once for the whole test project as an assembly attribute. [NonParallelizable] opts a fixture or test back out.
The bug: one Driver, several tests#
The obvious first attempt is a file Parallelism.cs with:
[assembly: Parallelizable(ParallelScope.All)]
[assembly: LevelOfParallelism(4)]
Run the suite and some tests fail with errors that have nothing to do with what they test:
Failed WaitsForVisibility [1 s]
1) System.ObjectDisposedException : Cannot access a disposed object.
Object name: 'System.Net.Http.HttpClient'.
2) TearDown : System.ObjectDisposedException : Cannot access a disposed object.
Failed TextAndPasswordAreSubmitted [680 ms]
1) OpenQA.Selenium.WebDriverException : The HTTP request to the remote WebDriver server for URL http://localhost:64662/session/60b640d5.../element timed out after 60 seconds.
By default NUnit creates one instance of each fixture class and runs all its tests on that instance. Driver is an instance property. With ParallelScope.All, two tests in WaitTests run at the same time on the same object: the second test's SetUp overwrites Driver with a new browser, the first test's TearDown quits it, and the second test then talks to a driver that has been disposed. Which tests fail changes from run to run, which makes this a textbook source of flaky tests.
ParallelScope.Fixtures would not show the bug, because tests inside one fixture still run one after another. That is a valid choice, but it caps parallelism at the number of fixture classes.
The fix: one fixture instance per test#
NUnit can create a new instance of the fixture class for every test instead:
// Run every test in parallel, each in a fresh instance of its fixture class,
// so the Driver property is never shared between two running tests.
[assembly: Parallelizable(ParallelScope.All)]
[assembly: FixtureLifeCycle(LifeCycle.InstancePerTestCase)]
[assembly: LevelOfParallelism(4)]
With LifeCycle.InstancePerTestCase, each test gets its own object, so its own Driver property and its own page object fields, and nothing in BaseTest or the test classes has to change. Three runs in a row of the 16-test suite all passed.
You will also see suites that keep the driver in a ThreadLocal<IWebDriver> (the usual approach in Java with TestNG). It works, but it is more code, and it breaks in subtle ways once tests use async/await, because an awaited continuation can resume on a different thread. In NUnit, InstancePerTestCase is the simpler tool. What never works is a static driver field: that is one browser shared by every test in the process.
After adding [assembly: Parallelizable(ParallelScope.All)], tests in the same class fail at random with ObjectDisposedException or "invalid session id". Each test calls Driver.Quit() in TearDown. What is the cause?
A single fixture instance means a single Driver property shared by every test running in it. One test's TearDown disposes the browser another test is still using. One instance per test gives each test its own property. Too few cores would make tests slow, not produce disposed-object errors.
How many workers?#
LevelOfParallelism sets the number of worker threads, and so the maximum number of browsers open at once. Without it NUnit uses the number of processors or 2, whichever is greater. Override it per run without touching code:
dotnet test -- NUnit.NumberOfTestWorkers=2
Note the space after --: everything after it goes to the NUnit adapter as a run setting. The same setting can live in a .runsettings file. On our machine, the 16 headless Chrome tests took 25 seconds with 1 worker and 7 to 9 seconds with 4.
Choose the number by where the browsers run:
- Locally, each browser costs real memory and CPU. Four to eight is a sensible start on a developer laptop; past that, tests start timing out because the machine is overloaded, not because the app is slow.
- On a grid, the browsers run remotely and the limit is your plan's number of parallel sessions. Set the worker count to roughly that number, lower if other pipelines share the account. A higher number does not finish sooner: the extra sessions wait in the queue, and once parallel plus queued sessions exceed what the plan allows, new sessions are refused outright. We saw exactly that on a shared account:
OpenQA.Selenium.UnsupportedOperationException : You are currently at the limit of the maximum allowed parallel + queued tests for your TestingBot plan. Your plan allows 2 parallel tests and 4 pending tests.
For our own grid runs, on an account shared with several other jobs, we used NUnit.NumberOfTestWorkers=1.
Your TestingBot plan allows 5 parallel sessions. A teammate sets NUnit.NumberOfTestWorkers=20 in CI to make the grid run faster. What happens?
The plan's parallel limit, not the worker count, decides how many browsers run. Extra workers only wait in the queue, and past the queue cap session creation is refused, which shows up as test errors in SetUp.
Keep tests independent#
Parallel runs expose any hidden dependency between tests, because the order and the timing are no longer fixed:
-
No test may depend on another test having run. NUnit 5 even deprecates
[Order]. If a test needs data, its ownSetUpcreates it. -
No shared mutable state in
staticfields: counters, lists of created users, a cached driver. -
Unique test data. Two tests that both register
test@example.comcollide when they run at the same time. Add aGuidor the test name. -
Mark the exceptions. A test that changes global settings of the app under test can be marked
[NonParallelizable]and runs on its own.
The flaky tests course covers how to find and fix the order and timing dependencies that parallel runs reveal, and cross-browser testing covers choosing the browser matrix that parallel runs make affordable. The last step is to run all of this automatically on every push.