Exit course
Chapter 8 of 10 Beginner 6 min read 2 quick checks Updated

Tags and parallel test runs with Pabot

On this page
  1. Tags
  2. Why robot runs one test at a time
  3. What parallel runs require
  4. Parallel runs on TestingBot

The suite now has ten tests across three files. Two practical questions follow: how do you run only some of them, and how do you run them faster? Tags answer the first, Pabot the second.

Tags#

A tag is a free-form label on a test. Test Tags in the Settings section tags every test in the file; [Tags] on a test adds more:

*** Settings ***
Documentation     The web form, written with user keywords.
Resource          ../resources/web_form.resource
Test Setup        Open The Web Form
Test Teardown     Close Test Browser
Test Tags         form

*** Test Cases ***
Submitting The Form Shows A Confirmation
    [Tags]    smoke
    Fill In The Form    Robot Framework
    Submit The Form
    Confirmation Should Be Shown

In tests/login.robot, add Test Tags login to Settings and [Tags] smoke to the valid-login test. (Test Tags replaced the older Force Tags setting; old suites still use the old name.)

Select tests by tag on the command line:

robot --outputdir results --include smoke tests
Valid Login Reaches The Secure Area                                   | PASS |
Tests.Login :: Data-driven login checks with a test template.         | PASS |
Submitting The Form Shows A Confirmation                              | PASS |
Tests.Web Form :: The web form, written with user keywords.           | PASS |
Tests                                                                 | PASS |
2 tests, 2 passed, 0 failed

--include and --exclude can be repeated and take patterns: smokeANDlogin matched one test in our suite, smokeORform four, and loginNOTsmoke the four failing-login rows. Wildcards work too (--include login*).

Tags that pay for themselves:

  • smoke for a handful of fast, critical tests to run on every commit, with the full suite nightly.
  • Feature or page tags (login, form) to run just the area you are changing.
  • A tag for known problems, so they can be excluded deliberately while being fixed, rather than deleted or commented out. Robot Framework also has a built-in robot:skip tag that marks a test as skipped in the report.

robot --dryrun --include smoke tests lists what a selection would run without opening a browser, which is a quick way to check a tag pattern before a long run.

The pipeline should run every login test except the slow ones tagged slow. Which command does that?

Include selects the login tests and exclude removes the slow ones from that set (loginNOTslow in a single pattern is equivalent). loginANDslow selects exactly the tests you wanted to leave out, and --test matches test names, not tags.

Why robot runs one test at a time#

robot runs tests strictly one after another. That is simple and predictable, but a browser test spends most of its time waiting for pages, and on a grid each test also waits for a remote browser to start. Ten tests at a few seconds each is fine; three hundred is not.

Pabot (installed in chapter 1) splits the suite over several robot processes and merges their results into one report:

pabot --processes 4 --testlevelsplit --outputdir results tests/web_form.robot tests/login.robot
PASSED Suites.Login.Wrong Password in 3.1 seconds
PASSED Suites.Login.Unknown User in 3.3 seconds
PASSED Suites.Login.Empty Username in 3.3 seconds
PASSED Suites.Login.Username Is Case Sensitive in 3.3 seconds
PASSED Suites.Web Form.Disabled Input Cannot Be Edited in 1.5 seconds
PASSED Suites.Web Form.The Dropdown Keeps The Chosen Option in 1.7 seconds
PASSED Suites.Web Form.Submitting The Form Shows A Confirmation in 2.0 seconds
PASSED Suites.Login.Valid Login Reaches The Secure Area in 2.9 seconds
8 tests, 8 passed, 0 failed, 0 skipped.
...
Total testing: 26.70 seconds
Elapsed time:  8.43 seconds

The same eight tests took about 17 seconds with plain robot on the same machine; with four processes they finished in under 9. (Pabot's own log lines also carry a timestamp and process ID, trimmed here.)

The two options that matter most:

  • --processes N is how many tests run at once. The default is the number of CPU cores, at least 2.
  • --testlevelsplit runs each test on its own. Without it Pabot splits by suite file, so all five login tests run in one process one after another. Suite-level splitting is the safer default when tests in a file share something set up once in a Suite Setup; test-level splitting is faster when every test is independent, as ours are.

Everything else, --variable, --include, --outputdir, is passed on to each robot process unchanged.

What parallel runs require#

Parallel runs expose any hidden dependency between tests. Ours run safely because of decisions made earlier in the course:

  • Each test opens its own browser in Test Setup and closes it in Test Teardown, so no two processes share a browser.
  • No test depends on another having run first. The valid-login test does not leave state for anything else.
  • Nothing writes to shared data. If your tests create users or orders, give each test unique data (a timestamp or a random suffix) instead of a fixed name two processes would fight over. Pabot's PabotLib offers locks and shared value sets for the rare case where tests really must coordinate.

The flaky tests course covers test isolation in more depth.

A suite file logs in once in Suite Setup and its tests reuse that browser. With pabot --testlevelsplit the tests start failing. Why?

Test-level splitting runs every test in a separate robot process, so anything one test relied on from a previous one is gone. Either make each test independent or keep the default suite-level split for that file.

Parallel runs on TestingBot#

On the grid, Pabot works exactly as locally; only the variables change:

export TB_BUILD=robot-course-1
pabot --processes 2 --testlevelsplit --outputdir results \
      --variable GRID:True --variable BROWSER:edge --variable PLATFORM:WIN11 \
      --include login tests

Set --processes to the number of parallel sessions your TestingBot plan allows, and lower it if other pipelines share the account. More processes than the plan allows do not make the run faster: according to the Robot Framework guide, sessions over the limit are queued for up to six minutes until a slot frees up. The account we used was shared with other jobs while writing this course, so we ran the five login tests on Edge with --processes 1; all five passed and appeared in one build.

Run the suite on several browsers by calling Pabot once per browser with the same TB_BUILD, so every session of the run is grouped in one build. The cross-browser testing course discusses which combinations are worth the time.