CircleCI Automated Testing

Get CircleCI up and running

Sign up at CircleCI and connect your project with CircleCI.

Both variables can also be stored in a CircleCI context, so several projects share the same TestingBot credentials.

The TestingBot CircleCI Orb

The TestingBot orb is published in the CircleCI registry as testingbot/testingbot. Its source lives on GitHub.

The orb takes care of the TestingBot Tunnel for you: it downloads the tunnel, starts it, waits until it is ready to accept traffic and shuts it down again when the job finishes. That makes it possible to test a web application that only exists inside the CircleCI job, such as a dev server on localhost, a staging environment, or any host behind your firewall.

The orb provides:

• A with_tunnel job that wraps your test steps and handles setup and teardown.
• Three commands, install_tunnel, start_tunnel and stop_tunnel, for when you want to control the tunnel inside your own job.
• A default executor based on cimg/openjdk, which ships the Java runtime the tunnel needs.

Quick start: the with_tunnel job

The fastest way to get started is the with_tunnel job. Pass it the steps you want to run, and the orb installs and starts the tunnel, runs your steps and always stops the tunnel afterwards, even when your tests fail.

version: 2.1

orbs:
  testingbot: testingbot/testingbot@1.0.0

workflows:
  test:
    jobs:
      - testingbot/with_tunnel:
          tunnel_identifier: circleci-<< pipeline.number >>
          steps:
            - run:
                name: Run cross-browser tests on TestingBot
                command: npm run test:e2e

Your credentials are read from the TB_KEY and TB_SECRET environment variables you configured above. If you store them under different names, point the key and secret parameters at those variable names.

Using the commands in your own job

When you need more control, for example because you want to build the application first or use a specific Docker image, use the individual commands instead:

version: 2.1

orbs:
  testingbot: testingbot/testingbot@1.0.0

jobs:
  e2e-tests:
    executor: testingbot/default
    steps:
      - checkout
      - testingbot/install_tunnel
      - testingbot/start_tunnel:
          tunnel_identifier: my-tunnel
      - run:
          name: Run Selenium tests through the tunnel
          command: npm run test:e2e
      - testingbot/stop_tunnel

workflows:
  test:
    jobs:
      - e2e-tests

stop_tunnel runs with when: always, so the tunnel is cleaned up even when an earlier step failed.

Pointing your tests at the tunnel

Once the tunnel is ready, it exposes a local Selenium relay on port 4445. Point your tests at http://localhost:4445/wd/hub instead of hub.testingbot.com, and traffic is routed through the tunnel to the TestingBot cloud. Use the se_port parameter if you need a different port.

import org.openqa.selenium.WebDriver;
import org.openqa.selenium.chrome.ChromeOptions;
import org.openqa.selenium.remote.RemoteWebDriver;
import java.net.URL;
import java.util.HashMap;
import java.util.Map;

String url = "http://localhost:4445/wd/hub";

ChromeOptions options = new ChromeOptions();
options.setPlatformName(System.getenv("SELENIUM_PLATFORM"));
options.setBrowserVersion(System.getenv("SELENIUM_BROWSER_VERSION"));

Map<String, Object> tbOptions = new HashMap<>();
tbOptions.put("build", "CircleCI build " + System.getenv("CIRCLE_BUILD_NUM"));
tbOptions.put("tunnelIdentifier", System.getenv("TB_TUNNEL_IDENTIFIER"));
options.setCapability("tb:options", tbOptions);

WebDriver driver = new RemoteWebDriver(new URL(url), options);

import os
from selenium import webdriver

url = "http://localhost:4445/wd/hub"

options = webdriver.ChromeOptions()
options.set_capability('platformName', os.environ.get('SELENIUM_PLATFORM', 'WIN11'))
options.set_capability('browserVersion', os.environ.get('SELENIUM_BROWSER_VERSION', 'latest'))
options.set_capability('tb:options', {
    'build': "CircleCI build {}".format(os.environ.get('CIRCLE_BUILD_NUM')),
    'tunnelIdentifier': os.environ.get('TB_TUNNEL_IDENTIFIER', 'my-tunnel')
})

driver = webdriver.Remote(command_executor=url, options=options)

const { Builder } = require('selenium-webdriver');

const url = 'http://localhost:4445/wd/hub';

const driver = await new Builder()
  .usingServer(url)
  .withCapabilities({
    browserName: process.env.SELENIUM_BROWSER || 'chrome',
    platformName: process.env.SELENIUM_PLATFORM || 'WIN11',
    browserVersion: process.env.SELENIUM_BROWSER_VERSION || 'latest',
    'tb:options': {
      build: `CircleCI build ${process.env.CIRCLE_BUILD_NUM}`,
      tunnelIdentifier: process.env.TB_TUNNEL_IDENTIFIER || 'my-tunnel'
    }
  })
  .build();

require 'selenium/webdriver'

url = "http://localhost:4445/wd/hub"

options = Selenium::WebDriver::Chrome::Options.new
options.add_option('platformName', ENV['SELENIUM_PLATFORM'] || 'WIN11')
options.add_option('browserVersion', ENV['SELENIUM_BROWSER_VERSION'] || 'latest')
options.add_option('tb:options', {
  'build' => "CircleCI build #{ENV['CIRCLE_BUILD_NUM']}",
  'tunnelIdentifier' => ENV['TB_TUNNEL_IDENTIFIER'] || 'my-tunnel'
})

browser = Selenium::WebDriver.for(:remote, url: url, options: options)

use Facebook\WebDriver\Remote\RemoteWebDriver;
use Facebook\WebDriver\Remote\DesiredCapabilities;

$url = 'http://localhost:4445/wd/hub';

$capabilities = DesiredCapabilities::chrome();
$capabilities->setCapability('platformName', getenv('SELENIUM_PLATFORM') ?: 'WIN11');
$capabilities->setCapability('browserVersion', getenv('SELENIUM_BROWSER_VERSION') ?: 'latest');
$capabilities->setCapability('tb:options', [
    'build' => 'CircleCI build ' . getenv('CIRCLE_BUILD_NUM'),
    'tunnelIdentifier' => getenv('TB_TUNNEL_IDENTIFIER') ?: 'my-tunnel'
]);

$driver = RemoteWebDriver::create($url, $capabilities);

using System;
using System.Collections.Generic;
using OpenQA.Selenium.Chrome;
using OpenQA.Selenium.Remote;

var url = "http://localhost:4445/wd/hub";

ChromeOptions options = new ChromeOptions();
options.PlatformName = Environment.GetEnvironmentVariable("SELENIUM_PLATFORM") ?? "WIN11";
options.BrowserVersion = Environment.GetEnvironmentVariable("SELENIUM_BROWSER_VERSION") ?? "latest";
options.AddAdditionalOption("tb:options", new Dictionary<string, object>
{
    ["build"] = $"CircleCI build {Environment.GetEnvironmentVariable("CIRCLE_BUILD_NUM")}",
    ["tunnelIdentifier"] = Environment.GetEnvironmentVariable("TB_TUNNEL_IDENTIFIER") ?? "my-tunnel"
});

IWebDriver driver = new RemoteWebDriver(new Uri(url), options);

When several tunnels run in parallel, each one needs its own identifier. Pass the same value in the tunnelIdentifier capability so your test is routed through the correct tunnel. See running multiple tunnels for more detail.

Setting the same build capability on every session in the workflow groups them into a single TestingBot build, which you can then show with a status badge or query from the REST API.

Orb reference

Job: with_tunnel

Runs your test steps with a TestingBot Tunnel active.

Command: install_tunnel

Downloads and installs the TestingBot Tunnel jar. Skips the download when the jar is already present.

Command: start_tunnel

Starts the tunnel in the background and waits until it is ready. Requires install_tunnel to have run first.

Command: stop_tunnel

Gracefully stops a tunnel started with start_tunnel. It takes no parameters and runs even when earlier steps failed.

Executor: default

A Docker executor based on cimg/openjdk, which provides the Java runtime the tunnel needs. Use the tag parameter (default 17.0) to pick a different image tag:

jobs:
  e2e-tests:
    executor:
      name: testingbot/default
      tag: "21.0"

Testing without a tunnel

If your application is publicly reachable, you do not need the tunnel at all. Point your tests directly at hub.testingbot.com and use whichever executor suits your test stack.

Pick your language below for a .circleci/config.yml file and the matching test code.

version: 2.1

jobs:
  test:
    parallelism: 3 # depends on how many concurrent sessions your TestingBot subscription has
    docker:
      - image: cimg/openjdk:17.0
    environment:
      TB_BUILD: "build No. $CIRCLE_BUILD_NUM for CircleCI"
      SELENIUM_PLATFORM: WIN11
      SELENIUM_BROWSER: chrome
      SELENIUM_BROWSER_VERSION: "latest"
    steps:
      - checkout
      - run: mvn -B test

workflows:
  test:
    jobs:
      - test

version: 2.1

jobs:
  test:
    parallelism: 3 # depends on how many concurrent sessions your TestingBot subscription has
    docker:
      - image: cimg/python:3.12
    environment:
      TB_BUILD: "build No. $CIRCLE_BUILD_NUM for CircleCI"
      SELENIUM_PLATFORM: WIN11
      SELENIUM_BROWSER: chrome
      SELENIUM_BROWSER_VERSION: "latest"
    steps:
      - checkout
      - run: pip install -r requirements.txt
      - run: pytest

workflows:
  test:
    jobs:
      - test

version: 2.1

jobs:
  test:
    parallelism: 3 # depends on how many concurrent sessions your TestingBot subscription has
    docker:
      - image: cimg/node:20.11
    environment:
      TB_BUILD: "build No. $CIRCLE_BUILD_NUM for CircleCI"
      SELENIUM_PLATFORM: WIN11
      SELENIUM_BROWSER: chrome
      SELENIUM_BROWSER_VERSION: "latest"
    steps:
      - checkout
      - run: npm ci
      - run: npm test

workflows:
  test:
    jobs:
      - test

version: 2.1
orbs:
  ruby: circleci/ruby@2.1.0
  node: circleci/node@5.2.0

jobs:
  test:
    parallelism: 3 # depends on how many concurrent sessions your TestingBot subscription has
    docker:
      - image: cimg/ruby:3.2-node
    environment:
      TB_BUILD: "build No. $CIRCLE_BUILD_NUM for CircleCI"
      SELENIUM_PLATFORM: WIN11
      SELENIUM_BROWSER: chrome
      SELENIUM_BROWSER_VERSION: "latest"
    steps:
      - checkout
      - ruby/install:
          version: '3.2'
      - ruby/install-deps
      - run: bundle exec cucumber

workflows:
  test:
    jobs:
      - test

version: 2.1

jobs:
  test:
    parallelism: 3 # depends on how many concurrent sessions your TestingBot subscription has
    docker:
      - image: cimg/php:8.3
    environment:
      TB_BUILD: "build No. $CIRCLE_BUILD_NUM for CircleCI"
      SELENIUM_PLATFORM: WIN11
      SELENIUM_BROWSER: chrome
      SELENIUM_BROWSER_VERSION: "latest"
    steps:
      - checkout
      - run: composer install --no-interaction
      - run: ./vendor/bin/phpunit

workflows:
  test:
    jobs:
      - test

version: 2.1

jobs:
  test:
    parallelism: 3 # depends on how many concurrent sessions your TestingBot subscription has
    docker:
      - image: mcr.microsoft.com/dotnet/sdk:8.0
    environment:
      TB_BUILD: "build No. $CIRCLE_BUILD_NUM for CircleCI"
      SELENIUM_PLATFORM: WIN11
      SELENIUM_BROWSER: chrome
      SELENIUM_BROWSER_VERSION: "latest"
    steps:
      - checkout
      - run: dotnet restore
      - run: dotnet test

workflows:
  test:
    jobs:
      - test

You can now use the environment variables, defined above and on CircleCI, in your code:

import org.openqa.selenium.WebDriver;
import org.openqa.selenium.chrome.ChromeOptions;
import org.openqa.selenium.remote.RemoteWebDriver;
import java.net.URL;
import java.util.HashMap;
import java.util.Map;

String url = String.format("https://%s:%s@hub.testingbot.com/wd/hub",
    System.getenv("TB_KEY"), System.getenv("TB_SECRET"));

ChromeOptions options = new ChromeOptions();
options.setPlatformName(System.getenv("SELENIUM_PLATFORM"));
options.setBrowserVersion(System.getenv("SELENIUM_BROWSER_VERSION"));

Map<String, Object> tbOptions = new HashMap<>();
tbOptions.put("build", System.getenv("TB_BUILD"));
options.setCapability("tb:options", tbOptions);

WebDriver driver = new RemoteWebDriver(new URL(url), options);

import os
from selenium import webdriver

url = "https://{}:{}@hub.testingbot.com/wd/hub".format(
    os.environ['TB_KEY'], os.environ['TB_SECRET'])

options = webdriver.ChromeOptions()
options.set_capability('platformName', os.environ.get('SELENIUM_PLATFORM', 'WIN11'))
options.set_capability('browserVersion', os.environ.get('SELENIUM_BROWSER_VERSION', 'latest'))
options.set_capability('tb:options', {
    'build': os.environ.get('TB_BUILD')
})

driver = webdriver.Remote(command_executor=url, options=options)

const { Builder } = require('selenium-webdriver');

const url = `https://${process.env.TB_KEY}:${process.env.TB_SECRET}@hub.testingbot.com/wd/hub`;

const driver = await new Builder()
  .usingServer(url)
  .withCapabilities({
    browserName: process.env.SELENIUM_BROWSER || 'chrome',
    platformName: process.env.SELENIUM_PLATFORM || 'WIN11',
    browserVersion: process.env.SELENIUM_BROWSER_VERSION || 'latest',
    'tb:options': {
      build: process.env.TB_BUILD
    }
  })
  .build();

require 'selenium/webdriver'

url = "https://#{ENV['TB_KEY']}:#{ENV['TB_SECRET']}@hub.testingbot.com/wd/hub"

options = Selenium::WebDriver::Chrome::Options.new
options.add_option('platformName', ENV['SELENIUM_PLATFORM'] || 'WIN11')
options.add_option('browserVersion', ENV['SELENIUM_BROWSER_VERSION'] || 'latest')
options.add_option('tb:options', {
  'build' => ENV['TB_BUILD']
})

browser = Selenium::WebDriver.for(:remote, url: url, options: options)

Before do |scenario|
  @browser = browser
end

at_exit do
  browser.quit
end

use Facebook\WebDriver\Remote\RemoteWebDriver;
use Facebook\WebDriver\Remote\DesiredCapabilities;

$url = sprintf('https://%s:%s@hub.testingbot.com/wd/hub',
    getenv('TB_KEY'), getenv('TB_SECRET'));

$capabilities = DesiredCapabilities::chrome();
$capabilities->setCapability('platformName', getenv('SELENIUM_PLATFORM') ?: 'WIN11');
$capabilities->setCapability('browserVersion', getenv('SELENIUM_BROWSER_VERSION') ?: 'latest');
$capabilities->setCapability('tb:options', [
    'build' => getenv('TB_BUILD')
]);

$driver = RemoteWebDriver::create($url, $capabilities);

using System;
using System.Collections.Generic;
using OpenQA.Selenium.Chrome;
using OpenQA.Selenium.Remote;

var url = $"https://{Environment.GetEnvironmentVariable("TB_KEY")}:{Environment.GetEnvironmentVariable("TB_SECRET")}@hub.testingbot.com/wd/hub";

ChromeOptions options = new ChromeOptions();
options.PlatformName = Environment.GetEnvironmentVariable("SELENIUM_PLATFORM") ?? "WIN11";
options.BrowserVersion = Environment.GetEnvironmentVariable("SELENIUM_BROWSER_VERSION") ?? "latest";
options.AddAdditionalOption("tb:options", new Dictionary<string, object>
{
    ["build"] = Environment.GetEnvironmentVariable("TB_BUILD")
});

IWebDriver driver = new RemoteWebDriver(new Uri(url), options);

More information

• TestingBot orb in the CircleCI registry
• Orb source code on GitHub
• TestingBot Tunnel documentation
• All tunnel command-line options
• Status badges