Robot Framework Browser library: Playwright on TestingBot
On this page
The Browser library gives Robot Framework the Playwright engine. Your tests keep the same shape (settings, test cases, user keywords), but the keywords underneath are different, and two of the problems from chapter 2 mostly go away: Playwright waits for elements to be ready before acting on them, and assertions retry until they pass or time out.
This chapter writes the web form tests again with the Browser library, runs them locally, and then on TestingBot's Playwright grid. They ran on Chrome and Firefox on Windows 11.
Browsers, contexts and pages#
The Browser library mirrors Playwright's three levels:
- a browser is the browser process (
New Browserlocally,Connect To Browserfor a remote one); - a context is an isolated profile inside it, with its own cookies, storage and viewport (
New Context); - a page is a tab (
New Page).
Starting a new context per test is cheap and gives every test a clean slate without starting a new browser.
Connecting to TestingBot#
TestingBot's Playwright endpoint is wss://cloud.testingbot.com/playwright. The browser you want is described by a capabilities query parameter holding URL-encoded JSON: browserName, browserVersion, platform and the same tb:options as in chapter 4.
That JSON contains your key and secret, which leads to the one rule of this section: never put credentials into Robot Framework variables. Robot logs every variable assignment at the default log level. Our first version built the capabilities in Robot syntax, with &{tb}= Create Dictionary key=%{TB_KEY} secret=%{TB_SECRET}, and when we searched the run's output.xml for the secret, it was there four times:
&{tb} = { key=... | secret=... | name=Submitting The Form Shows A Confirmation | ... }
The assignment logs the dictionary, and the next one logs the encoded endpoint that contains it again. The same goes for passing credentials as keyword arguments, which the log also records. output.xml and log.html are exactly the files CI uploads as an artifact.
The Browser library has a second log. By default it writes playwright-log.txt into the output directory, and that file recorded the full endpoint, secret included, even when the endpoint was built in Python. Import the library with enable_playwright_debug=disabled to turn it off.
So the connection moves into libraries/TestingBot.py, next to the Selenium options, where the secret only ever exists inside Python. The complete file, with the tb:options dictionary now shared by both functions:
import json
import os
from datetime import timedelta
from urllib.parse import quote
from Browser.utils.data_types import SupportedBrowsers
from robot.libraries.BuiltIn import BuiltIn
from selenium import webdriver
OPTIONS = {
"chrome": webdriver.ChromeOptions,
"firefox": webdriver.FirefoxOptions,
"edge": webdriver.EdgeOptions,
"safari": webdriver.SafariOptions,
}
ENGINES = {"chrome": "chromium", "edge": "chromium", "firefox": "firefox", "safari": "webkit"}
def _tb_options(name, build):
return {
"key": os.environ["TB_KEY"],
"secret": os.environ["TB_SECRET"],
"name": name,
"build": build,
}
def testingbot_options(browser, platform, version="latest", name="", build=""):
"""Returns Selenium options for one TestingBot browser."""
options = OPTIONS[browser.lower()]()
options.browser_version = version
options.platform_name = platform
options.set_capability("tb:options", _tb_options(name, build))
return options
def connect_to_testingbot(browser, platform, version="latest", name="", build=""):
"""Connects the Browser library to a TestingBot Playwright browser."""
capabilities = {
"browserName": browser.lower(),
"browserVersion": version,
"platform": platform,
"tb:options": _tb_options(name, build),
}
endpoint = ("wss://cloud.testingbot.com/playwright?capabilities="
+ quote(json.dumps(capabilities)))
# Called from Python, so the endpoint (with the secret) never reaches the log.
BuiltIn().get_library_instance("Browser").connect_to_browser(
endpoint, SupportedBrowsers[ENGINES[browser.lower()]], timeout=timedelta(minutes=2))
What connect_to_testingbot does:
-
ENGINESmaps the browser you ask TestingBot for to the Playwright engine that talks to it: Chrome and Edge usechromium, Firefoxfirefox, Safariwebkit. -
BuiltIn().get_library_instance("Browser")returns the Browser library instance the suite already imported, and the function calls itsConnect To Browserkeyword directly from Python. Called this way, the endpoint is not a keyword argument and not an assignment, so it never appears in the log. A function whose name starts with an underscore, like_tb_options, is not exposed as a keyword. -
timeout=timedelta(minutes=2): the default is 30 seconds. Our first Firefox run failed withbrowserType.connect: Timeout 30000ms exceededwhile the remote browser was still starting, and with two minutes both Firefox tests passed. Chrome connected well within the default. On a busy account a connection can also wait in the queue for a free slot, which counts against the same timeout.
After this change, and with enable_playwright_debug=disabled in the resource file below, we ran the suite on the grid again and searched every file in the output directory for the key and the secret: no matches.
The resource file#
resources/playwright.resource:
*** Settings ***
Documentation Starts a Playwright browser, locally or on TestingBot.
Library Browser enable_playwright_debug=disabled
Library ../libraries/TestingBot.py
*** Variables ***
${BROWSER} chrome
${GRID} ${FALSE}
${PLATFORM} WIN11
${VERSION} latest
&{ENGINES} chrome=chromium edge=chromium firefox=firefox safari=webkit
*** Keywords ***
Start Browser
IF ${GRID}
Connect To TestingBot ${BROWSER} ${PLATFORM} ${VERSION}
... name=${TEST NAME} build=%{TB_BUILD=robot-local}
ELSE
New Browser ${ENGINES}[${BROWSER}] headless=True
END
New Context viewport={'width': 1920, 'height': 1080}
Stop Browser
IF ${GRID} Run Keyword And Ignore Error Report Status To TestingBot
Close Browser
Report Status To TestingBot
${passed}= Evaluate $TEST_STATUS == 'PASS'
&{arguments}= Create Dictionary passed=${passed} reason=${TEST MESSAGE}
&{command}= Create Dictionary action=setSessionStatus arguments=${arguments}
${json}= Evaluate json.dumps($command) modules=json
Evaluate JavaScript ${None} () => {} arg=testingbot_executor: ${json}
-
enable_playwright_debug=disabledstops the library from writingplaywright-log.txt, which would otherwise contain the connection URL with your credentials. -
Locally,
New Browserstarts the engine from the&{ENGINES}dictionary;${ENGINES}[${BROWSER}]looks a value up. -
New Context viewport=...fixes the viewport, for the same reason chapter 2 set a window size. -
Run Keyword And Ignore Errorkeeps the status report best-effort: if the connection orNew Pagefailed in setup, there is no page to report through, and the report's own error would otherwise be added to the real failure. -
Evaluateruns a Python expression.$TEST_STATUS(without braces) passes the variable's value as a Python object rather than pasting it into the expression as text.
The Playwright options reference lists everything else the endpoint accepts. By default TestingBot uses the same Playwright version as the client that connects, so you do not have to keep two versions in step.
Reporting the result#
As with Selenium, the grid cannot see your assertions. On the Playwright endpoint the result is sent as a testingbot_executor: string passed as the argument of an evaluate call; TestingBot intercepts it rather than running it in the page. Evaluate JavaScript ${None} () => {} arg=... does exactly that: no selector, an empty function and the command as its argument. setSessionStatus takes passed and an optional reason, which here is Robot's failure message. The same command is documented in the Playwright Test guide.
Safari tests on TestingBot's Playwright grid fail as soon as they connect, while Chrome works. Connect To Browser is called with browser=chromium. What is wrong?
The engine on the client side must match the browser on the other side of the connection: chromium for Chrome and Edge, firefox for Firefox, webkit for Safari. That is what the ENGINES lookup is for.
The tests#
tests/playwright/web_form.robot:
*** Settings ***
Documentation The web form again, with the Browser library (Playwright).
Resource ../../resources/playwright.resource
Test Setup Start Browser
Test Teardown Stop Browser
*** Variables ***
${FORM URL} https://www.selenium.dev/selenium/web/web-form.html
*** Test Cases ***
Submitting The Form Shows A Confirmation
New Page ${FORM URL}
Get Title == Web form
Fill Text id=my-text-id Robot Framework
Select Options By select[name=my-select] label Two
Click button[type=submit]
Get Text id=message == Received!
Get Title == Web form - target page
Disabled Input Cannot Be Edited
New Page ${FORM URL}
Get Element States input[name=my-disabled] contains disabled
Compare it with chapter 3:
-
Selectors are Playwright selectors. A plain value is CSS; prefixes like
id=,text=andxpath=pick another engine. -
Assertions are built into getters.
Get Text id=message == Received!reads the text and compares it, and the Browser library retries the comparison until it passes or the timeout (10 seconds by default) runs out. There is no separateWait Untilfamily to remember. -
Actions wait too.
Clickwaits until the button is visible, enabled and stable. After the submit there is no explicit wait at all:Get Texton the next page simply retries until the message is there.
Run it locally, then on the grid:
robot --outputdir results tests/playwright
robot --outputdir results --variable GRID:True --variable BROWSER:firefox tests/playwright
Submitting The Form Shows A Confirmation | PASS |
------------------------------------------------------------------------------
Disabled Input Cannot Be Edited | PASS |
------------------------------------------------------------------------------
Playwright.Web Form :: The web form again, with the Browser librar... | PASS |
2 tests, 2 passed, 0 failed
In the dashboard both sessions show the Robot test name, the build name and a passed status.
You port a SeleniumLibrary test to the Browser library. Which version of the "message appears after submit" check is idiomatic there?
With an assertion operator the getter retries until the value matches. Reading the value first and comparing it with Should Be Equal checks only once, so it brings back the timing problem. Wait Until Page Contains is a SeleniumLibrary keyword.
Which library should your suite use?#
For a new suite, the Browser library's automatic waiting removes a whole class of flaky failures, and Playwright's contexts make test isolation cheap. SeleniumLibrary remains the safe choice when you have an existing suite, need a browser or version that only a WebDriver grid offers, or test mobile web through Appium. Both run on TestingBot, so you can also mix them: one suite per library, each with its own resource file, in the same project.
The Playwright course covers Playwright itself in depth.