Tags and parallel test runs with Pabot
On this page
The suite now has ten tests across three files. Two practical questions follow: how do you run only some of them, and how do you run them faster? Tags answer the first, Pabot the second.
Tags#
A tag is a free-form label on a test. Test Tags in the Settings section tags every test in the file; [Tags] on a test adds more:
*** Settings ***
Documentation The web form, written with user keywords.
Resource ../resources/web_form.resource
Test Setup Open The Web Form
Test Teardown Close Test Browser
Test Tags form
*** Test Cases ***
Submitting The Form Shows A Confirmation
[Tags] smoke
Fill In The Form Robot Framework
Submit The Form
Confirmation Should Be Shown
In tests/login.robot, add Test Tags login to Settings and [Tags] smoke to the valid-login test. (Test Tags replaced the older Force Tags setting; old suites still use the old name.)
Select tests by tag on the command line:
robot --outputdir results --include smoke tests
Valid Login Reaches The Secure Area | PASS |
Tests.Login :: Data-driven login checks with a test template. | PASS |
Submitting The Form Shows A Confirmation | PASS |
Tests.Web Form :: The web form, written with user keywords. | PASS |
Tests | PASS |
2 tests, 2 passed, 0 failed
--include and --exclude can be repeated and take patterns: smokeANDlogin matched one test in our suite, smokeORform four, and loginNOTsmoke the four failing-login rows. Wildcards work too (--include login*).
Tags that pay for themselves:
-
smokefor a handful of fast, critical tests to run on every commit, with the full suite nightly. -
Feature or page tags (
login,form) to run just the area you are changing. -
A tag for known problems, so they can be excluded deliberately while being fixed, rather than deleted or commented out. Robot Framework also has a built-in
robot:skiptag that marks a test as skipped in the report.
robot --dryrun --include smoke tests lists what a selection would run without opening a browser, which is a quick way to check a tag pattern before a long run.
The pipeline should run every login test except the slow ones tagged slow. Which command does that?
Include selects the login tests and exclude removes the slow ones from that set (loginNOTslow in a single pattern is equivalent). loginANDslow selects exactly the tests you wanted to leave out, and --test matches test names, not tags.
Why robot runs one test at a time#
robot runs tests strictly one after another. That is simple and predictable, but a browser test spends most of its time waiting for pages, and on a grid each test also waits for a remote browser to start. Ten tests at a few seconds each is fine; three hundred is not.
Pabot (installed in chapter 1) splits the suite over several robot processes and merges their results into one report:
pabot --processes 4 --testlevelsplit --outputdir results tests/web_form.robot tests/login.robot
PASSED Suites.Login.Wrong Password in 3.1 seconds
PASSED Suites.Login.Unknown User in 3.3 seconds
PASSED Suites.Login.Empty Username in 3.3 seconds
PASSED Suites.Login.Username Is Case Sensitive in 3.3 seconds
PASSED Suites.Web Form.Disabled Input Cannot Be Edited in 1.5 seconds
PASSED Suites.Web Form.The Dropdown Keeps The Chosen Option in 1.7 seconds
PASSED Suites.Web Form.Submitting The Form Shows A Confirmation in 2.0 seconds
PASSED Suites.Login.Valid Login Reaches The Secure Area in 2.9 seconds
8 tests, 8 passed, 0 failed, 0 skipped.
...
Total testing: 26.70 seconds
Elapsed time: 8.43 seconds
The same eight tests took about 17 seconds with plain robot on the same machine; with four processes they finished in under 9. (Pabot's own log lines also carry a timestamp and process ID, trimmed here.)
The two options that matter most:
-
--processes Nis how many tests run at once. The default is the number of CPU cores, at least 2. -
--testlevelsplitruns each test on its own. Without it Pabot splits by suite file, so all five login tests run in one process one after another. Suite-level splitting is the safer default when tests in a file share something set up once in aSuite Setup; test-level splitting is faster when every test is independent, as ours are.
Everything else, --variable, --include, --outputdir, is passed on to each robot process unchanged.
What parallel runs require#
Parallel runs expose any hidden dependency between tests. Ours run safely because of decisions made earlier in the course:
-
Each test opens its own browser in
Test Setupand closes it inTest Teardown, so no two processes share a browser. - No test depends on another having run first. The valid-login test does not leave state for anything else.
-
Nothing writes to shared data. If your tests create users or orders, give each test unique data (a timestamp or a random suffix) instead of a fixed name two processes would fight over. Pabot's
PabotLiboffers locks and shared value sets for the rare case where tests really must coordinate.
The flaky tests course covers test isolation in more depth.
A suite file logs in once in Suite Setup and its tests reuse that browser. With pabot --testlevelsplit the tests start failing. Why?
Test-level splitting runs every test in a separate robot process, so anything one test relied on from a previous one is gone. Either make each test independent or keep the default suite-level split for that file.
Parallel runs on TestingBot#
On the grid, Pabot works exactly as locally; only the variables change:
export TB_BUILD=robot-course-1
pabot --processes 2 --testlevelsplit --outputdir results \
--variable GRID:True --variable BROWSER:edge --variable PLATFORM:WIN11 \
--include login tests
Set --processes to the number of parallel sessions your TestingBot plan allows, and lower it if other pipelines share the account. More processes than the plan allows do not make the run faster: according to the Robot Framework guide, sessions over the limit are queued for up to six minutes until a slot frees up. The account we used was shared with other jobs while writing this course, so we ran the five login tests on Edge with --processes 1; all five passed and appeared in one build.
Run the suite on several browsers by calling Pabot once per browser with the same TB_BUILD, so every session of the run is grouped in one build. The cross-browser testing course discusses which combinations are worth the time.