Run Selenium Python tests on the TestingBot grid

Everything so far ran on your own machine: one operating system, whichever browser versions you have installed, and as much parallelism as your CPU allows. A Selenium grid runs the browsers on remote machines instead. Your tests stay exactly the same; only the fixture changes, from webdriver.Chrome() to webdriver.Remote(grid_url, options=...). This chapter makes that switch behind a --grid flag, using the TestingBot grid. The same webdriver.Remote code works with a self-hosted Selenium Grid; only the URL and the vendor-specific options differ.

Everything in this chapter was run on TestingBot: the final suite ran as 21 sessions (7 tests on Chrome and Firefox on Windows 11 and Safari on macOS Sonoma), all reported as passed, plus deliberate failures to check that failed status and screenshots come through.

What changes and what doesn't#

Because chapter 4 put driver creation in the fixture, the switch touches only conftest.py.

Capabilities and tb:options#

A remote session is requested with the same options classes you use locally (ChromeOptions, FirefoxOptions, EdgeOptions, SafariOptions) plus two standard W3C capabilities, which the options classes expose as properties:

• options.browser_version, such as "latest" or a specific major version;
• options.platform_name, the operating system. On TestingBot these are names such as WIN11 for Windows 11 or SONOMA for macOS Sonoma.

Vendor settings go in a namespaced capability. TestingBot's is tb:options, which carries your key and secret, name (the test name shown in the dashboard) and build (which groups the tests of one run). The capabilities generator produces the exact values for any browser and OS, and test options lists everything else tb:options accepts, such as screen resolution, time zone, or turning off video recording, which is on by default.

The final conftest.py#

import os
import re
from pathlib import Path

import pytest
from selenium import webdriver

GRID_URL = "https://hub.testingbot.com/wd/hub"

# browser name -> (options class, TestingBot platform)
BROWSERS = {
    "chrome": (webdriver.ChromeOptions, "WIN11"),
    "firefox": (webdriver.FirefoxOptions, "WIN11"),
    "edge": (webdriver.EdgeOptions, "WIN11"),
    "safari": (webdriver.SafariOptions, "SONOMA"),
}


def pytest_addoption(parser):
    parser.addoption("--browser", action="append", choices=sorted(BROWSERS),
                     help="browser to run on; repeat for several (default: chrome)")
    parser.addoption("--headless", action="store_true", help="run local browsers headless")
    parser.addoption("--grid", action="store_true", help="run on the TestingBot grid")


def pytest_generate_tests(metafunc):
    # Every test that uses the driver runs once per --browser value.
    if "browser" in metafunc.fixturenames:
        browsers = metafunc.config.getoption("browser") or ["chrome"]
        metafunc.parametrize("browser", browsers)


@pytest.hookimpl(wrapper=True, tryfirst=True)
def pytest_runtest_makereport(item, call):
    report = yield
    # Remember the outcome of each phase so fixtures can read it during teardown.
    setattr(item, "rep_" + report.when, report)
    return report


def local_driver(browser, headless):
    if browser == "firefox":
        options = webdriver.FirefoxOptions()
        if headless:
            options.add_argument("-headless")
        return webdriver.Firefox(options=options)
    if browser == "chrome":
        options = webdriver.ChromeOptions()
        if headless:
            options.add_argument("--headless=new")
            options.add_argument("--window-size=1920,1080")
        return webdriver.Chrome(options=options)
    raise pytest.UsageError(f"{browser} is only configured for --grid runs")


def remote_driver(browser, test_name):
    options_class, platform = BROWSERS[browser]
    options = options_class()
    options.browser_version = "latest"
    options.platform_name = platform
    options.set_capability("tb:options", {
        "key": os.environ["TB_KEY"],
        "secret": os.environ["TB_SECRET"],
        "name": test_name,
        "build": os.environ.get("TB_BUILD", "local"),
    })
    return webdriver.Remote(command_executor=GRID_URL, options=options)


@pytest.fixture
def driver(request, browser):
    config = request.config
    if config.getoption("grid"):
        driver = remote_driver(browser, request.node.name)
    else:
        driver = local_driver(browser, config.getoption("headless"))

    yield driver

    call = getattr(request.node, "rep_call", None)
    failed = call is None or call.failed
    try:
        if failed:
            save_screenshot(driver, request.node.nodeid)
        if config.getoption("grid"):
            status = "failed" if failed else "passed"
            driver.execute_script(f"tb:test-result={status}")
            print(f"TestingBot session {driver.session_id}: {status}")
    finally:
        driver.quit()


def save_screenshot(driver, nodeid):
    folder = Path("artifacts")
    folder.mkdir(exist_ok=True)
    worker = os.environ.get("PYTEST_XDIST_WORKER", "main")
    name = re.sub(r"[^\w.-]+", "_", nodeid)
    path = folder / f"{name}-{worker}.png"
    driver.save_screenshot(str(path))
    print(f"screenshot: {path}")

What is new compared with chapter 5:

• --grid chooses between local_driver and remote_driver. Without it, nothing about the local workflow changes.
• BROWSERS maps the --browser names to an options class and a platform. Safari only exists on macOS, so it gets SONOMA; the others run on Windows 11. Edge and Safari are only configured for grid runs, and asking for them locally fails with a clear UsageError rather than silently starting Chrome.
• Credentials come from the TB_KEY and TB_SECRET environment variables, never from code or pytest.ini. Find both in the TestingBot member area; in CI, store them as secrets. os.environ[...] (rather than .get) fails loudly when they are missing.
• name is the pytest test name including its parameters, such as test_dropdown_selection_is_submitted[firefox], so each dashboard entry maps to one line of your pytest output.
• build comes from TB_BUILD, so all sessions from one run are grouped together.

Reporting pass or fail#

The grid only sees WebDriver commands. It cannot know that your assert failed; from its side the session simply ended. Unless you tell it, the dashboard cannot show which tests passed. The fixture reports the outcome during teardown, before quit(), through the JavaScript executor:

driver.execute_script("tb:test-result=passed")   # or failed

TestingBot recognises tb: commands and handles them on the grid side instead of running them as page JavaScript. The same mechanism accepts tb:test-name=... and others, all listed with examples in annotating tests. It needs no extra package, but it must run while the session is open.

The alternative is the REST API, through the testingbotclient package from requirements.txt. Use it when you want to report after the session has closed, or from outside the test process:

from testingbotclient import TestingBotClient


@pytest.fixture
def driver(request):
    ...  # create the remote driver as above
    session_id = driver.session_id
    yield driver
    driver.quit()

    call = getattr(request.node, "rep_call", None)
    TestingBotClient().tests.update_test(session_id, passed=bool(call and call.passed))

TestingBotClient() with no arguments reads the same TB_KEY and TB_SECRET variables. The pytest guide shows the other fields update_test accepts. Pick one of the two mechanisms; using both is harmless but redundant.

Run it#

export TB_KEY=your-key
export TB_SECRET=your-secret
export TB_BUILD="course-$(date +%Y%m%d-%H%M)"
pytest --grid --browser chrome --browser firefox --browser safari -n 2

21 passed in 129.10s (0:02:09)

Seven tests times three browsers, two at a time. Open the build in the TestingBot dashboard to see every session with its name, status, video and logs.

Parallel limits#

Every grid caps how many sessions can run at once. On TestingBot the limit comes with your plan, and according to the docs, sessions beyond it are queued for up to 6 minutes and start as soon as a slot is free. The queue has a cap too. When we ran this suite with -n 6 on an account whose plan allows 2 parallel and 4 queued tests, while other jobs were using the same account, some sessions were refused outright:

selenium.common.exceptions.WebDriverException: Message: You are currently at the limit of the maximum allowed parallel + queued tests for your TestingBot plan. Your plan allows 2 parallel tests and 4 pending tests.

Set -n to roughly your parallel limit, and lower it if other pipelines share the account. A higher number does not make the run faster; it only adds queue time and, past the queue cap, errors.

Testing a site that is not public#

A cloud browser cannot reach localhost:8000 or a staging server behind your firewall. The TestingBot tunnel solves this: start it on a machine that can reach the site, then point webdriver.Remote at the tunnel instead of the public hub, by default http://localhost:4445/wd/hub when the tunnel runs on the same machine as the tests. The pytest guide shows the exact options and the tunnel documentation covers installation; the local website testing course goes into it in depth.

Continuous integration with GitHub Actions#

Store TB_KEY and TB_SECRET as repository secrets (Settings, Secrets and variables, Actions), then add .github/workflows/ui-tests.yml:

name: UI tests
on: [push, pull_request]

jobs:
  selenium:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: "3.13"
          cache: pip
      - run: pip install -r requirements.txt
      - name: Run the suite on TestingBot
        env:
          TB_KEY: ${{ secrets.TB_KEY }}
          TB_SECRET: ${{ secrets.TB_SECRET }}
          TB_BUILD: github-${{ github.run_id }}-${{ github.run_attempt }}
        run: >
          pytest --grid --browser chrome --browser firefox -n 2
          --junitxml=junit.xml --html=report.html --self-contained-html
      - uses: actions/upload-artifact@v4
        if: always()
        with:
          name: test-results
          path: |
            junit.xml
            report.html
            artifacts/

TB_BUILD built from the run ID and attempt groups every session of one workflow run into one TestingBot build, and a re-run gets a new one. if: always() uploads the reports and screenshots even when tests fail, which is exactly when you need them. The GitHub Actions guide adds the tunnel action for private sites and pull request status checks.

Debugging a remote failure#

Remote failures feel harder because you cannot watch the browser. In practice you have more evidence than locally:

• The screenshot fixture still works. save_screenshot is a WebDriver command, so it captures the remote browser.
• The session ID is printed by the fixture on every grid run, for example TestingBot session 1014ca63...: failed, and appears in the failure output next to the screenshot path. It ties a pytest failure to the exact session in the dashboard.
• The video and logs of that session show what the browser did around the failing command.

Most "only fails on the grid" problems turn out to be a slower page load that an explicit wait would have covered, a window size different from your laptop's, or a genuine browser difference you would never have seen running Chrome alone, which is the reason to run on several browsers in the first place.