Bitbucket Pipelines WebDriver Testing

Prerequisites

• A Bitbucket Cloud repository with Pipelines enabled (Repository settings → Pipelines → Settings → Enable Pipelines).
• A bitbucket-pipelines.yml file in the root of that repository.
• An existing Selenium or Appium test suite.
• A TestingBot key and secret, available in the TestingBot member area.

Credentials

Your tests read the TestingBot key and secret from the TESTINGBOT_KEY and TESTINGBOT_SECRET environment variables.

Add them in Repository settings → Pipelines → Repository variables. Tick the Secured checkbox for the secret: Bitbucket then stores the value encrypted and replaces it with the variable name everywhere it would otherwise appear in the build log.

Bitbucket offers three scopes, which are resolved in this order:

Configuration

Create a bitbucket-pipelines.yml in the root of your repository. Pick an image that matches your test suite, install the dependencies and run the tests:

image: node:20

pipelines:
  default:
    - step:
        name: Selenium tests on TestingBot
        caches:
          - node
        script:
          - npm ci
          - npm run test:e2e

image: python:3.12

pipelines:
  default:
    - step:
        name: Selenium tests on TestingBot
        caches:
          - pip
        script:
          - pip install -r requirements.txt
          - pytest --junitxml=test-results/results.xml

image: maven:3.9-eclipse-temurin-21

pipelines:
  default:
    - step:
        name: Selenium tests on TestingBot
        caches:
          - maven
        script:
          - mvn -B clean test

image: ruby:3.4

pipelines:
  default:
    - step:
        name: Selenium tests on TestingBot
        caches:
          - bundler
        script:
          - bundle install
          - bundle exec rspec

The TESTINGBOT_KEY and TESTINGBOT_SECRET repository variables are injected automatically, so there is nothing to add to the script for authentication.

Example Test

Your test reads the credentials from the environment and sends the TestingBot capabilities in the tb:options object:

const webdriver = require('selenium-webdriver');
const chrome = require('selenium-webdriver/chrome');

const build = `bitbucket-${process.env.BITBUCKET_BUILD_NUMBER}`;

async function runBitbucketTest () {
  let options = new chrome.Options();
  options.set('platformName', 'WIN11');
  options.set('browserVersion', 'latest');
  options.set('tb:options', {
    'key': process.env.TESTINGBOT_KEY,
    'secret': process.env.TESTINGBOT_SECRET,
    'name': 'Bitbucket Pipelines Test',
    'build': build,
    'tunnel-identifier': build
  });

  let driver = new webdriver.Builder()
    .usingServer('https://hub.testingbot.com/wd/hub')
    .withCapabilities(options)
    .build();
  await driver.get('http://localhost:8080');
  console.log(await driver.getTitle());
  await driver.quit();
}
runBitbucketTest();

import os
from selenium import webdriver
from selenium.webdriver.chrome.options import Options

build = "bitbucket-%s" % os.environ.get('BITBUCKET_BUILD_NUMBER')

options = Options()
options.set_capability('platformName', 'WIN11')
options.set_capability('browserVersion', 'latest')
options.set_capability('tb:options', {
  'key': os.environ['TESTINGBOT_KEY'],
  'secret': os.environ['TESTINGBOT_SECRET'],
  'name': 'Bitbucket Pipelines Test',
  'build': build,
  'tunnel-identifier': build
})

driver = webdriver.Remote(
    command_executor='https://hub.testingbot.com/wd/hub',
    options=options
)
driver.get("http://localhost:8080")
print(driver.title)
driver.quit()

import org.openqa.selenium.WebDriver;
import org.openqa.selenium.chrome.ChromeOptions;
import org.openqa.selenium.remote.RemoteWebDriver;

import java.net.URL;
import java.util.HashMap;
import java.util.Map;

public class BitbucketTest {

  public static final String KEY = System.getenv("TESTINGBOT_KEY");
  public static final String SECRET = System.getenv("TESTINGBOT_SECRET");
  public static final String URL = "https://hub.testingbot.com/wd/hub";

  public static void main(String[] args) throws Exception {
    String build = "bitbucket-" + System.getenv("BITBUCKET_BUILD_NUMBER");

    ChromeOptions options = new ChromeOptions();
    options.setCapability("browserVersion", "latest");
    options.setCapability("platformName", "WIN11");

    Map<String, Object> tbOptions = new HashMap<>();
    tbOptions.put("key", KEY);
    tbOptions.put("secret", SECRET);
    tbOptions.put("name", "Bitbucket Pipelines Test");
    tbOptions.put("build", build);
    tbOptions.put("tunnel-identifier", build);
    options.setCapability("tb:options", tbOptions);

    WebDriver driver = new RemoteWebDriver(new URL(URL), options);
    driver.get("http://localhost:8080");

    System.out.println(driver.getTitle());

    driver.quit();
  }
}

require 'selenium-webdriver'

build = "bitbucket-#{ENV['BITBUCKET_BUILD_NUMBER']}"

options = Selenium::WebDriver::Chrome::Options.new
options.add_option('platformName', 'WIN11')
options.add_option('browserVersion', 'latest')
options.add_option('tb:options', {
  'key' => ENV['TESTINGBOT_KEY'],
  'secret' => ENV['TESTINGBOT_SECRET'],
  'name' => 'Bitbucket Pipelines Test',
  'build' => build,
  'tunnel-identifier' => build
})

driver = Selenium::WebDriver.for(
  :remote,
  url: "https://hub.testingbot.com/wd/hub",
  options: options)
driver.navigate.to "http://localhost:8080"
puts driver.title
driver.quit

Leave out the tunnel-identifier capability when you are testing a publicly reachable website without a tunnel.

Grouping Builds

By passing the build capability, all sessions started by one pipeline run are grouped into a single TestingBot build, which gives you one combined pass or fail result per commit instead of a long list of unrelated tests.

Bitbucket exposes a number of default variables you can use to name your builds and tests:

A readable convention is to use the build number for the build capability and the branch plus commit for the test name, so you can trace a failing session straight back to a commit.

Privately Hosted Websites

If the website under test is not publicly reachable, a staging environment behind a firewall or a server started by the build itself, run a TestingBot Tunnel in the pipeline. There are two ways to do this in Bitbucket Pipelines.

Option 1: start the tunnel in the step

The tunnel is a Java application, so use an image that has a JRE available or install one first. Start the tunnel in the background, wait for its ready file, then run your tests:

image: node:20

pipelines:
  default:
    - step:
        name: Selenium tests on TestingBot
        caches:
          - node
        script:
          - apt-get update && apt-get install -y --no-install-recommends default-jre-headless unzip
          - npm ci
          - curl -sS -O https://testingbot.com/downloads/testingbot-tunnel.zip
          - unzip -o -q testingbot-tunnel.zip
          - |
            java -jar testingbot-tunnel-*.jar $TESTINGBOT_KEY $TESTINGBOT_SECRET \
              --tunnel-identifier bitbucket-$BITBUCKET_BUILD_NUMBER \
              --readyfile tunnel.ready \
              --logfile tunnel.log &
          - timeout 180 bash -c 'until [ -f tunnel.ready ]; do sleep 1; done'
          - npm run start:staging &
          - npm run test:e2e
        after-script:
          - pkill -f testingbot-tunnel || true
        artifacts:
          - tunnel.log

A few things worth pointing out:

• The --readyfile flag touches a file as soon as the tunnel is connected. Waiting for that file is more reliable than a fixed sleep, because your tests never start before the tunnel is up. Keep both the ready file and the log inside the clone directory, otherwise Bitbucket cannot pick the log up as an artifact.
• The --tunnel-identifier flag names the tunnel, so concurrent pipelines each get their own. Pass the same value as the tunnel-identifier capability in your tests.
• after-script runs whether the step passed or failed, which makes it the right place to stop the tunnel. Without it the step keeps running until the pipeline times out.
• The testingbot-tunnel.zip download always contains the latest tunnel version, which is why the example uses a wildcard for the jar name.

Option 2: run the tunnel as a service container

Bitbucket service containers share their network adapter with the build container, so a tunnel started as a service is reachable on localhost and can in turn reach any server your step starts. This keeps your script free of tunnel plumbing and works with an image that has no Java, using the official testingbot/tunnel image from Docker Hub:

image: node:20

definitions:
  services:
    testingbot-tunnel:
      image: testingbot/tunnel
      memory: 1024
      variables:
        TESTINGBOT_KEY: $TESTINGBOT_KEY
        TESTINGBOT_SECRET: $TESTINGBOT_SECRET

pipelines:
  default:
    - step:
        name: Selenium tests on TestingBot
        services:
          - testingbot-tunnel
        caches:
          - node
        script:
          - npm ci
          - npm run start:staging &
          - timeout 180 bash -c 'until (echo > /dev/tcp/127.0.0.1/4445) 2>/dev/null; do sleep 2; done'
          - npm run test:e2e

With this setup you do not need a tunnel identifier. Point your tests at the Selenium relay the tunnel exposes on http://localhost:4445/wd/hub instead of at https://hub.testingbot.com/wd/hub: the relay forwards your sessions to TestingBot and routes the traffic back through the tunnel. See the tunnel command line options for all available flags.

Parallel Steps

Cross browser suites are the obvious candidate for Bitbucket's parallel steps: each step runs on its own build container at the same time, so the wall clock time of the pipeline is that of the slowest browser rather than the sum of all of them.

pipelines:
  default:
    - parallel:
        - step:
            name: Chrome on Windows 11
            script:
              - BROWSER=chrome PLATFORM=WIN11 npm run test:e2e
        - step:
            name: Safari on macOS
            script:
              - BROWSER=safari PLATFORM=SONOMA npm run test:e2e
        - step:
            name: Firefox on Windows 11
            script:
              - BROWSER=firefox PLATFORM=WIN11 npm run test:e2e

Use the same build capability in every step, so the three steps still show up as one TestingBot build. If your steps each start a tunnel, give them a unique identifier using bitbucket-$BITBUCKET_BUILD_NUMBER-$BITBUCKET_PARALLEL_STEP.

Test Reports

Bitbucket parses JUnit style XML and shows failing tests on a Tests tab next to the build log. It scans these directories automatically, up to three levels deep:

• ./**/test-results/**/*.xml
• ./**/test-reports/**/*.xml
• ./**/surefire-reports/**/*.xml
• ./**/failsafe-reports/**/*.xml
• ./**/TestResults/**/*.xml

If your test runner writes its report somewhere else, declare it explicitly:

- step:
    name: Selenium tests on TestingBot
    script:
      - npm run test:e2e
    artifacts:
      upload:
        - name: "e2e test reports"
          type: "test-reports"
          paths:
            - "reports/junit/*.xml"

This gives you the failure summary inside Bitbucket, while the video, screenshots, WebDriver log and network traffic of every session stay available on TestingBot.

Pull Request Reports

Bitbucket Code Insights lets a pipeline attach a report to a commit, which then shows up in the sidebar of every pull request containing that commit. It is a good way to surface a link to your TestingBot build without leaving the review flow.

The quickest way is our TestingBot Report pipe, which reads the TestingBot build and writes the report for you. Put it in after-script, so it also runs when your tests fail:

- step:
    name: Selenium tests on TestingBot
    script:
      - npm run test:e2e
    after-script:
      - pipe: testingbot/testingbot-report-pipe:0.1.0
        variables:
          TESTINGBOT_KEY: $TESTINGBOT_KEY
          TESTINGBOT_SECRET: $TESTINGBOT_SECRET

The pipe looks up the build named bitbucket-$BITBUCKET_BUILD_NUMBER, which is what the build capability in the example test above already sets. It publishes the pass and fail counts plus the total duration, and adds one annotation per failing test, each linking to the session on TestingBot. Set FAIL_ON_FAILED_TESTS: 'true' if you want the pipe itself to fail the step.

Without the pipe

You can also write the report yourself. Inside a pipeline you do not need any credentials for this: Pipelines runs an authenticating proxy on localhost:29418 that signs the request for you.

- step:
    name: Selenium tests on TestingBot
    script:
      - npm run test:e2e
    after-script:
      - |
        if [ "$BITBUCKET_EXIT_CODE" = "0" ]; then RESULT=PASSED; else RESULT=FAILED; fi
        cat > report.json <<EOF
        {
          "title": "TestingBot",
          "details": "Cross browser tests for build $BITBUCKET_BUILD_NUMBER",
          "report_type": "TEST",
          "reporter": "TestingBot",
          "result": "$RESULT",
          "link": "https://testingbot.com/members/builds",
          "data": [
            { "title": "Build", "type": "TEXT", "value": "bitbucket-$BITBUCKET_BUILD_NUMBER" }
          ]
        }
        EOF
        curl --proxy http://localhost:29418 --request PUT \
          "http://api.bitbucket.org/2.0/repositories/$BITBUCKET_WORKSPACE/$BITBUCKET_REPO_SLUG/commit/$BITBUCKET_COMMIT/reports/testingbot-$BITBUCKET_BUILD_NUMBER" \
          --header 'Content-Type: application/json' \
          --data @report.json

The report id must be unique per commit, which is why the example includes the build number. A report carries at most 10 data entries. If you want per file detail, add annotations to the report with the path and line number of each failing test, and Bitbucket renders them inline in the diff.

Verify the Results

Open the Pipelines section of your repository to follow the run, and the Tests tab of a step for the failure summary.

At the same time your sessions appear in the TestingBot member area, grouped per build, with video, screenshots, WebDriver logs and network traffic for every test.

Troubleshooting

• Authentication failed: the pipeline could not read your credentials. Repository variables are not available to pipelines triggered from a fork, and a variable defined at workspace level is overridden by a repository variable of the same name. Check for a stale copy in both places.
• The secret shows up as $TESTINGBOT_SECRET in the log: that is expected, it means the variable is secured. The real value is still passed to your test.
• The tunnel never becomes ready: make sure a Java runtime is present in the image and inspect /tmp/tunnel.log, which the example above uploads as an artifact.
• Tests time out on a local URL: verify that the tunnel-identifier capability matches the --tunnel-identifier passed to the tunnel, and that your local server is already listening when the tests start.
• The step hangs after the tests: a background tunnel is still running. Stop it in after-script as shown above.
• The step runs out of memory: services share the memory budget of the step. Lower the memory of the tunnel service, or switch the step to size: 2x.
• Sessions queue instead of starting: your parallel steps exceed the concurrency of your TestingBot plan. Reduce the number of parallel steps or upgrade the plan.