Exit course
Chapter 9 of 10 Beginner 6 min read 2 quick checks Updated

Robot Framework reports, CI and test status on TestingBot

On this page
  1. The three output files
  2. The exit code
  3. Combining and post-processing results
  4. Run the suite in GitHub Actions
  5. Pass and fail on TestingBot, recapped

A suite earns its keep when it runs on every change and its failures are quick to understand. This chapter covers the files every run produces, the exit code your CI reads, rerunning failures, and a GitHub Actions workflow that runs the suite on TestingBot.

The three output files#

Every robot run writes three files into --outputdir:

File What it is Who reads it
output.xml the complete machine-readable result rebot, CI plugins, Pabot's merge step
log.html every suite, test and keyword with arguments, timings, log messages and screenshots you, when something failed
report.html a summary with statistics by tag and by suite, linking into the log anyone who wants the overview

When a test fails, open log.html and expand the failing test. The failing keyword is highlighted and SeleniumLibrary's failure screenshot (selenium-screenshot-1.png, also in the output folder) is embedded right below it. If your keywords are well named, the path to the failure reads like a sentence: Open The Web Form, Fill In The Form, Submit The Form, Confirmation Should Be Shown failed on Title Should Be.

The log shows the arguments of every keyword, so be careful what goes into it. Input Password hides the value; your key and secret are only inside tb:options built in Python, not passed as keyword arguments. Adding --loglevel TRACE logs even more, including return values, which helps debugging but also shows more data.

The exit code#

robot exits with the number of failed tests (capped at 250), so 0 means everything passed. CI systems treat any non-zero exit code as a failed step, so you do not need to parse anything to fail the build. A run of the web form suite together with the deliberately failing test from chapter 4 exited with code 1.

Combining and post-processing results#

rebot turns one or more output.xml files into a log and report without running anything. Two uses come up constantly.

One report for several browser runs. Run each browser with its own output file and a suite name, then combine them:

robot --outputdir results --output chrome.xml --name Chrome --variable BROWSER:chrome --include smoke tests
robot --outputdir results --output firefox.xml --name Firefox --variable BROWSER:firefox --include smoke tests
rebot --outputdir results --name "Smoke on two browsers" --xunit xunit.xml results/chrome.xml results/firefox.xml

The combined report has one top-level suite per browser. --xunit xunit.xml also writes a JUnit-style XML file, the format most CI systems display as test results. Both robot and pabot accept --xunit directly too.

Rerunning only the failures.

robot --outputdir results tests
robot --outputdir results --output rerun.xml --rerunfailed results/output.xml tests
rebot --outputdir results --merge results/output.xml results/rerun.xml

--rerunfailed selects the tests that failed in the given output, and --merge writes a report where the rerun results replace the original ones, with a note that the test was rerun. Both runs must use the same root suite (here the tests folder); rebot refuses to merge outputs from different ones.

Use this to separate a genuine bug from a hiccup, not to make a red build green. A test that passes only on the rerun is a flaky test that needs fixing; the flaky tests course shows how to find out why.

The nightly job reruns failures automatically, and the merged report has been green for weeks. The same three tests appear as "rerun" almost every night. What should happen?

The merge records which tests needed a rerun precisely so you can see this pattern. A test that regularly fails first time either has a timing problem or catches a real intermittent defect; more reruns only hide it better.

Run the suite in GitHub Actions#

Store your TestingBot key and secret as repository secrets named TB_KEY and TB_SECRET (Settings, Secrets and variables, Actions). Then add .github/workflows/robot.yml:

name: Robot Framework tests
on: [push, pull_request]

jobs:
  robot:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: "3.13"
          cache: pip
      - uses: actions/setup-node@v4
        with:
          node-version: 22
      - run: pip install -r requirements.txt
      - run: rfbrowser init --skip-browsers
      - name: Run the suite on TestingBot
        env:
          TB_KEY: ${{ secrets.TB_KEY }}
          TB_SECRET: ${{ secrets.TB_SECRET }}
          TB_BUILD: github-${{ github.run_id }}-${{ github.run_attempt }}
        run: >
          pabot --processes 2 --testlevelsplit --outputdir results --xunit xunit.xml
          --variable GRID:True --variable BROWSER:chrome --variable PLATFORM:WIN11
          tests
      - uses: actions/upload-artifact@v4
        if: always()
        with:
          name: robot-results
          path: results/

Notes on the choices:

  • rfbrowser init --skip-browsers installs the Browser library's Node dependencies but no browsers: every browser runs on TestingBot, so the job does not need any.
  • TB_BUILD built from the run ID and attempt groups every session of one workflow run into one TestingBot build; a re-run of the workflow gets its own build. The resource files read it with %{TB_BUILD=robot-local}.
  • --processes 2: match your plan's parallel limit, as in chapter 7.
  • if: always() uploads the log, report and screenshots even when tests fail, which is exactly when you need them. Without it, a failing test step skips the upload.

For a site that only exists inside the CI job (started in an earlier step), add the TestingBot Tunnel action and point the suite at the tunnel; the GitHub Actions guide shows the setup.

Robot tests fail in the GitHub Actions job, but the robot-results artifact with the log is missing, so nobody can see why. What is the likely cause?

GitHub Actions skips later steps after a failed step unless they say otherwise. Robot always writes the log and report; the upload step must run regardless of the test result.

Pass and fail on TestingBot, recapped#

Your CI reads Robot's exit code; the TestingBot dashboard needs its own signal. Both resource files send it in the teardown, where ${TEST STATUS} is known:

  • SeleniumLibrary (chapter 4): Execute Javascript tb:test-result=passed or failed, before the browser closes.
  • Browser library (chapter 5): Evaluate JavaScript with a testingbot_executor: setSessionStatus command.

With both in place, the build in the dashboard matches the Robot report test for test: the same names, the same pass and fail, plus a video for every session. When a test fails in CI, the session ID in the console output or the test name takes you straight to the recording of what the browser did.

That completes the suite. The last chapter is a short knowledge check.