Exit course
Chapter 9 of 12 Intermediate 7 min read 2 quick checks Updated

Parallel Runs, Reports and CI on TestingBot

On this page
  1. Parallel scenarios with the JUnit Platform
  2. The same scenarios on several browsers
  3. Reports
  4. Running in GitHub Actions
  5. Sites that are not public

A BDD suite only helps if it runs on every change and its results are easy to read. This chapter makes the Java project from chapters 3 to 6 run scenarios in parallel, run the same scenarios on several browsers, produce reports a whole team can open, and run in GitHub Actions against the TestingBot grid with each session marked passed or failed.

Parallel scenarios with the JUnit Platform#

Cucumber-JVM runs scenarios one after another unless you tell it otherwise. With the JUnit Platform engine, parallel execution is switched on in src/test/resources/junit-platform.properties:

cucumber.execution.parallel.enabled=true
cucumber.execution.parallel.config.strategy=fixed
cucumber.execution.parallel.config.fixed.parallelism=2
cucumber.execution.parallel.config.fixed.max-pool-size=2

The unit of parallelism is the scenario: two scenarios from the same feature can run at the same time, and each row of a Scenario Outline counts as a scenario. The fixed strategy runs exactly the number of threads you ask for; max-pool-size stops the pool from growing beyond that.

Nothing else in the project changes, and that is the payoff of chapter 5. PicoContainer creates a fresh Browser, SignInPage and SignInSteps for every scenario, so each thread has its own WebDriver and no state is shared. A project that kept the driver in a static field would now have two scenarios typing into the same browser.

Locally, with headless Chrome, the four scenarios (three for signing in, one for signing out) took 11.7 seconds with one thread and 4.2 seconds with four. Any configuration parameter can be overridden for one run with a system property, which is handy when the grid allows fewer sessions than your laptop:

mvn test -Dcucumber.execution.parallel.config.fixed.parallelism=1 \
         -Dcucumber.execution.parallel.config.fixed.max-pool-size=1

Parallel limits on the grid#

On TestingBot, the number of sessions that run at the same time is limited by your plan. According to the docs, sessions beyond the limit are queued (for up to 6 minutes) and start when a slot frees up; when the queue is full as well, new sessions are refused with an HTTP 429 "You are currently at the limit of the maximum allowed parallel + queued tests". Set parallelism to about your plan's limit: more threads do not finish sooner, they just wait.

After enabling parallel execution, scenarios start failing randomly with steps typing into the wrong page. The suite passed when run sequentially. What is the most likely cause?

Parallel scenarios each need their own browser. A static driver is shared by every thread; per-scenario objects from the DI container are not. Queuing makes runs slower, not wrong.

The same scenarios on several browsers#

Cross-browser coverage is a property of the run, not of the scenarios, so it does not belong in the feature files (no @chrome tags). The DriverFactory reads the browser and platform from system properties, so each browser is one Maven invocation:

export TB_KEY=your-key
export TB_SECRET=your-secret
mvn test -Dgrid=testingbot -Dbrowser=chrome  -Dplatform=WIN11   -Dbuild=certificate-cucumber-bdd
mvn test -Dgrid=testingbot -Dbrowser=firefox -Dplatform=SEQUOIA -Dbuild=certificate-cucumber-bdd

Both runs use the same build, so the dashboard groups all eight sessions (four scenarios on two browsers) together. When the course ran these commands (with parallelism set to 1), every session in the build passed, among them:

Scenario Browser Session
Tom signs in with his own password Chrome 151, Windows 11 3530dee60d96-4a5e11ef84db-9ff99f642ac7-179456237146-4507345
Tom signs out Chrome 151, Windows 11 80c4f60f456b-209d22127a81-14f9db195bf5-179456228810-89122475
Wrong credentials are rejected (one outline row) Chrome 151, Windows 11 c992aeb85978-3084aa43ae2c-a7e8edc22ecc-179456234535-61673068
Tom signs in with his own password Firefox 154, macOS Sequoia 577c404568da-17432a370528-e447c7c6f7dc-179455766828-11620812
Wrong credentials are rejected (one outline row) Firefox 154, macOS Sequoia 92aa48238bb9-3853e6763829-329cc7d715bc-179455764622-77141610

Every session was reported as passed by the @After hook from chapter 5. Browser and OS names come from the capabilities generator; the cross-browser testing course helps choose which combinations are worth paying for.

Reports#

The runner from chapter 3 already writes three outputs:

Plugin Output For
pretty Scenarios and steps in the console Developers watching a run
html:target/cucumber.html One self-contained HTML file with features, scenarios, steps, attachments Anyone; attach it to the CI run
junit:target/cucumber.xml JUnit XML CI servers that show test results natively

The HTML report is the one to share with the business side: it shows the feature descriptions and scenarios as written, with failed steps highlighted and the screenshots attached by the @AfterStep hook embedded inline. Add message:target/cucumber.ndjson if you want the raw stream of Cucumber messages that other report tools consume.

For the browser side, the TestingBot dashboard keeps a video, screenshots and the command log of every session, named after the scenario. When a scenario fails only on one browser, open its session and watch the moment the step failed.

Running in GitHub Actions#

.github/workflows/cucumber.yml runs the suite on every pull request, on two browsers at the same time:

name: cucumber

on:
  pull_request:
  push:
    branches: [main]

jobs:
  scenarios:
    runs-on: ubuntu-latest
    strategy:
      fail-fast: false
      matrix:
        include:
          - browser: chrome
            platform: WIN11
          - browser: firefox
            platform: SEQUOIA
    steps:
      - uses: actions/checkout@v7
      - uses: actions/setup-java@v6
        with:
          distribution: temurin
          java-version: '21'
          cache: maven
      - name: Run scenarios on TestingBot
        env:
          TB_KEY: ${{ secrets.TB_KEY }}
          TB_SECRET: ${{ secrets.TB_SECRET }}
        run: >
          mvn -B test
          -Dgrid=testingbot
          -Dbrowser=${{ matrix.browser }}
          -Dplatform=${{ matrix.platform }}
          -Dbuild=github-${{ github.run_id }}
          -Dcucumber.filter.tags="not @wip"
      - name: Keep the Cucumber report
        if: always()
        uses: actions/upload-artifact@v7
        with:
          name: cucumber-${{ matrix.browser }}-${{ matrix.platform }}
          path: target/cucumber.html
  • Secrets: add TB_KEY and TB_SECRET under the repository's Settings, Secrets and variables, Actions. They never appear in code or logs.
  • The matrix starts one job per browser. Both jobs share the build name github-<run id>, so one pipeline run is one build on TestingBot. fail-fast: false keeps the Firefox job running when Chrome fails, so you see both results.
  • if: always() uploads the HTML report when the tests failed too, which is when you need it most.
  • Exit codes: Maven fails the job when any scenario fails, undefined steps included. Nothing extra is needed to make a red scenario block a merge.

The workflow file was checked with actionlint. Its Maven command is the grid command used above plus batch mode (-B) and a tag filter.

Each matrix job sets -Dbuild=github-${{ github.run_id }}. What does that give you on TestingBot?

The run ID is the same for all jobs of one workflow run, so every session lands in one build. Pass/fail still comes from the @After hook, not from the build name.

Sites that are not public#

A cloud browser cannot open localhost:8080 or a staging server behind your firewall. Start TestingBot Tunnel on a machine that can reach the site and point the remote driver at the tunnel instead of the public hub; by default that is http://localhost:4445/wd/hub when the tunnel runs on the same machine as the tests. The local website testing course covers the tunnel step by step, including in CI.