Parallel Runs, Suites and Running a Subset
On this page
The cloud run in chapter 6 started twelve sessions: four spec files on three browsers. Run one after another, that took about eight minutes. Run side by side, it takes as long as the slowest file. This chapter explains how WebdriverIO splits a run into workers, how to set the parallelism, and how to run only the part of the suite you need right now.
Workers: one per spec file and capability#
The runner's unit of work is a spec file in a capability. For each pair it starts a worker process, which opens one browser session, runs every test in that file, and closes the session. In the reporter output the worker id #2-1 means capability index 2 (Safari), spec index 1 (login.e2e.ts).
Three consequences follow:
- Tests in one file run one after another in the same browser session. Tests in different files never share a session and may run at the same time.
- Splitting a large spec file into several smaller ones is how you make a suite faster in parallel. A 40-test file is always one worker.
- Nothing is shared between files except what you set up again in each (a login, test data). Write every spec file as if it might run alone, because with
--specit will.
A suite has one spec file with 60 tests and runs on 2 browsers. You raise maxInstances from 2 to 10, but the run is no faster. Why?
WebdriverIO parallelises by spec file and capability. With one file there is nothing more to run in parallel per browser, whatever the limit. Splitting the file into several creates more workers.
Setting the parallelism#
Two settings cap how many workers run at once:
-
maxInstancesis the limit for the whole run. -
maxInstancesPerCapability(or'wdio:maxInstances'inside one capability) is the limit per browser, useful when one configuration is scarcer than the others.
Locally, the right number depends on your CPU and memory; each worker is a browser. On a cloud grid it should match the number of parallel sessions your plan allows. TestingBot queues a limited number of session requests beyond that limit; past the queue it refuses new ones straight away. One of the runs for this course hit exactly that, because other jobs were using the same account at the time:
WebDriverError: You are currently at the limit of the maximum allowed parallel + queued tests
for your TestingBot plan. Your plan allows 2 parallel tests and 4 pending tests.
WebdriverIO retried the request connectionRetryCount times (3 in the course config) within a couple of seconds, gave up, and reported the spec file as failed without running a single test. A failure like that says nothing about your application. Keep maxInstances at or below the plan's limit and the queue never fills.
The cloud config reads the value from an environment variable so you can change it without editing the file:
maxInstances: Number(process.env.TB_PARALLEL ?? 2),
The course suite was run with TB_PARALLEL=1 because the account it used was shared with other jobs (which is also how it ran into the limit above). With your own plan's limit the sessions run in waves: at TB_PARALLEL=4 the twelve sessions take three waves, at TB_PARALLEL=12 one wave, as long as the slowest spec file. The parallel testing calculator estimates how many parallel sessions a suite needs to finish in a target time.
Your TestingBot plan allows 5 parallel sessions. A teammate sets maxInstances: 50 "to be safe". Several spec files now fail before any test runs, with "at the limit of the maximum allowed parallel + queued tests". What is going on?
Session requests beyond the parallel limit queue only up to a point. Set maxInstances to the plan's limit so requests wait in the runner instead of being refused by the grid. No test code ran, so the specs are not at fault.
Grouping files into one worker#
Occasionally two files are cheap and you would rather pay for one session than two. Put them in a nested array:
specs: [
['./test/specs/waiting.e2e.ts', './test/specs/assertions.e2e.ts'],
'./test/specs/login.e2e.ts'
]
The inner array runs as one worker, in one session, one file after the other; the reporter shows both files under the same session id. Use it sparingly: grouped files lose their isolation from each other.
Suites#
A suite is a named list of spec files. Add a suites block to wdio.conf.ts:
specs: ['./test/specs/**/*.ts'],
suites: {
smoke: ['./test/specs/login.e2e.ts'],
waits: ['./test/specs/waiting.e2e.ts', './test/specs/assertions.e2e.ts']
},
Run a suite with --suite:
npx wdio run wdio.conf.ts --suite waits
[chrome 157.0.8103.78 mac #0-1] » test/specs/assertions.e2e.ts
[chrome 157.0.8103.78 mac #0-0] » test/specs/waiting.e2e.ts
Spec Files: 2 passed, 2 total (100% completed) in 00:00:14
A common split is a smoke suite that runs on every push and the full specs glob that runs nightly or before a release. Because the cloud config spreads baseConfig, the suites work there too: npm run wdio:tb -- --suite smoke.
Running a subset from the command line#
The flags combine:
# one file
npx wdio run wdio.conf.ts --spec test/specs/login.e2e.ts
# a suite plus one more file
npx wdio run wdio.conf.ts --suite smoke --spec test/specs/selectors.e2e.ts
# everything except one file
npx wdio run wdio.conf.ts --exclude test/specs/waiting.e2e.ts
# tests whose title matches, inside the selected files
npx wdio run wdio.conf.ts --spec test/specs/login.e2e.ts --mochaOpts.grep "wrong password"
--spec and --suite add up; --exclude removes files from whatever was selected. --mochaOpts.grep filters individual tests by title, so only the "shows an error for a wrong password" test runs in the last example. Any config option can be overridden this way with a dotted path.
Retries, with care#
specFileRetries reruns a failed spec file in a fresh worker, and Mocha's retries option reruns a failed test. They can keep a pipeline green while you investigate a known flaky test, and they can just as easily hide a real bug that only shows up some of the time. If you enable them, make sure retried tests are visible in the report, and treat a retry as a defect to fix. The flaky tests course covers how to find out why a test is flaky.