Exit course
Chapter 5 of 10 Beginner 8 min read 2 quick checks Updated

Run Robot Framework tests on TestingBot

On this page
  1. What the grid needs to know
  2. Build the options in Python
  3. Switch between local and grid
  4. Run it
  5. Tell TestingBot whether the test passed
  6. Sites that are not public

So far every browser ran on your machine: one operating system and whatever browser versions you happen to have. A remote grid runs the browser somewhere else. For SeleniumLibrary the change is small: Open Browser gets a remote_url and a set of capabilities describing the browser you want. This chapter adds that behind a GRID variable, so the same suite runs locally by default and on TestingBot with one switch.

Everything here ran on TestingBot: the web form suite on Chrome and Firefox on Windows 11, the login suite from chapter 6 on Safari on macOS Sequoia and Edge on Windows 11, plus a deliberately failing test to check that failures are reported as failures.

What the grid needs to know#

A remote session is requested with the same browser options classes Selenium uses locally, plus a few capabilities:

  • browserName, set by the options class (ChromeOptions, FirefoxOptions, EdgeOptions, SafariOptions);
  • browserVersion: latest, latest-1, or a specific major version;
  • platformName: the operating system, such as WIN11 for Windows 11 or SEQUOIA for macOS Sequoia;
  • tb:options: TestingBot's own settings, including your key and secret, the test name shown in the dashboard and the build that groups the tests of one run.

The capabilities generator produces exact values for every browser and platform, and test options lists everything else tb:options accepts, such as screen resolution or time zone.

Build the options in Python#

Capabilities can be built in Robot syntax with Call Method and Create Dictionary, but a few lines of Python are easier to read, and they keep your key and secret out of Robot's log. That is a rule worth keeping everywhere: never put credentials into Robot variables or keyword arguments, because Robot Framework logs assignments and arguments to output.xml and log.html (chapter 5 shows this happening). Create libraries/TestingBot.py:

import os

from selenium import webdriver

OPTIONS = {
    "chrome": webdriver.ChromeOptions,
    "firefox": webdriver.FirefoxOptions,
    "edge": webdriver.EdgeOptions,
    "safari": webdriver.SafariOptions,
}


def testingbot_options(browser, platform, version="latest", name="", build=""):
    """Returns Selenium options for one TestingBot browser."""
    options = OPTIONS[browser.lower()]()
    options.browser_version = version
    options.platform_name = platform
    options.set_capability("tb:options", {
        "key": os.environ["TB_KEY"],
        "secret": os.environ["TB_SECRET"],
        "name": name,
        "build": build,
    })
    return options

Imported as a library, the module's public functions become keywords: testingbot_options is called as TestingBot Options, and keyword arguments work as named arguments. The key and secret come from the environment, never from a file in the repository. os.environ[...] fails loudly with a KeyError if they are missing, which is better than a confusing authentication error later.

Switch between local and grid#

Extend resources/browser.resource:

*** Settings ***
Documentation     Opens and closes the browser, locally or on TestingBot.
Library           SeleniumLibrary    timeout=10s
Library           ../libraries/TestingBot.py

*** Variables ***
${BROWSER}        chrome
${GRID}           ${FALSE}
${PLATFORM}       WIN11
${VERSION}        latest
${HUB}            https://hub.testingbot.com/wd/hub

*** Keywords ***
Open Test Browser
    [Arguments]    ${url}
    IF    ${GRID}
        ${options}=    TestingBot Options    ${BROWSER}    ${PLATFORM}    ${VERSION}
        ...    name=${TEST NAME}    build=%{TB_BUILD=robot-local}
        Open Browser    ${url}    ${BROWSER}    remote_url=${HUB}    options=${options}
        ${session}=    Get Session Id
        Log To Console    \nTestingBot session ${session}
    ELSE
        Open Browser    ${url}    headless${BROWSER}
        Set Window Size    1920    1080
    END

Close Test Browser
    IF    ${GRID}
        ${result}=    Set Variable If    '${TEST STATUS}' == 'PASS'    passed    failed
        Run Keyword And Ignore Error    Execute Javascript    tb:test-result=${result}
    END
    Close All Browsers

What is new:

  • IF ${GRID} picks the branch. --variable GRID:True on the command line turns the grid on; Robot Framework evaluates True as a Python boolean.
  • ... continues a long line on the next one.
  • name=${TEST NAME} gives every session the Robot test name, so a dashboard entry maps to exactly one line of your report.
  • build=%{TB_BUILD=robot-local} reads the build name from the TB_BUILD environment variable, with robot-local as the default.
  • Get Session Id returns the WebDriver session ID, which is also TestingBot's test ID. Printing it ties a failure in your report to the session, video and logs in the dashboard.
  • No window size on the grid: remote sessions get a full-size screen; the screen-resolution option in tb:options changes it.

The test files do not change at all.

A teammate wants to commit ${TB_SECRET} abc123... to the Variables section so "everyone can run on the grid". What is the better approach?

Anything committed is visible to everyone with repository access and stays in the history after it is removed. Environment variables keep secrets out of the code; CI systems store them encrypted. Moving the secret to another committed file or into the URL does not change that.

Run it#

Find your key and secret in the TestingBot member area, then:

export TB_KEY=your-key
export TB_SECRET=your-secret
export TB_BUILD=robot-course-1
robot --outputdir results --variable GRID:True \
      --variable BROWSER:chrome --variable PLATFORM:WIN11 tests/web_form.robot
==============================================================================
Web Form :: The web form, written with user keywords.
==============================================================================
Submitting The Form Shows A Confirmation
TestingBot session 6b4e416c8079-d3c44ca2d53e-94edc5a832e3-179455636980-06780703
| PASS |
------------------------------------------------------------------------------
The Dropdown Keeps The Chosen Option
TestingBot session 5473da2637ef-854e326d911f-88dd39b69980-179455638140-02451200
| PASS |
------------------------------------------------------------------------------
Disabled Input Cannot Be Edited
TestingBot session 794803a699a8-2b0c36fc8393-1250f7aab6b1-179455639364-84453619
| PASS |
------------------------------------------------------------------------------
Web Form :: The web form, written with user keywords.                 | PASS |
3 tests, 3 passed, 0 failed
==============================================================================

Change the variables to change the browser: --variable BROWSER:firefox for Firefox on Windows 11, or --variable BROWSER:safari --variable PLATFORM:SEQUOIA for Safari, which only exists on macOS. Each session shows up in the dashboard under the build name, with its video, screenshots and WebDriver log.

Tell TestingBot whether the test passed#

The grid only sees WebDriver commands. It cannot know that Title Should Be failed; from its side the session simply ended. Unless you report the result, the dashboard cannot show which tests passed.

Close Test Browser does that in the teardown, where Robot Framework sets ${TEST STATUS} to PASS or FAIL. Execute Javascript tb:test-result=passed looks like JavaScript, but TestingBot intercepts tb: commands on the grid and records the result instead of running them in the page. Annotating tests lists the other tb: commands. Two details matter:

  • It must run before Close All Browsers: the command travels over the WebDriver session, so the session has to be open.
  • It is wrapped in Run Keyword And Ignore Error, so a session that already crashed does not turn a clear failure into a second, confusing teardown error.

To check it, we ran a test that is meant to fail, on Firefox:

Deliberately Failing Check
    Title Should Be    Not the web form
Deliberately Failing Check
TestingBot session b7ee2b4c5890-143a238fc1d6-a43bc4fab5b3-179455642090-13732081
| FAIL |
Title should have been 'Not the web form' but was 'Web form'.

and the TestingBot API confirmed what the dashboard showed: the passing sessions with "success": true, this one with "success": false.

The TestingBot Robot Framework guide shows an alternative: a small Python keyword that sends the status through the REST API with the session ID. That also works after the browser has closed. Whichever you use, the credentials are read from the environment inside Python, never passed in as a keyword argument. Use one of the two mechanisms, not both.

Robot's report shows two failures, but every session in the TestingBot dashboard has no result. The teardown is Close All Browsers followed by Execute Javascript tb:test-result=${result}. What is wrong?

The tb: command is sent over the WebDriver session. Once the browser is closed there is no session to send it on. ${TEST STATUS} exists precisely in test teardowns, and the command works on every browser.

Sites that are not public#

A cloud browser cannot reach http://localhost:8000 or a staging server behind your firewall. TestingBot Tunnel opens a route back: start it on a machine that can reach the site and point ${HUB} at the tunnel, by default http://localhost:4445/wd/hub, as shown in the Robot Framework guide. The local website testing course explains how the tunnel works.

The same suite now runs anywhere. The next chapter does the same with the Browser library.