AWS CodePipeline Automated Testing

Prerequisites

• An AWS account with permission to create CodePipeline and CodeBuild projects.
• A repository with your tests, connected to CodePipeline as a source.
• A buildspec.yml file in the root of that repository.
• A TestingBot key and secret, available in the TestingBot member area.

Credentials

Your tests read the TestingBot key and secret from the TESTINGBOT_KEY and TESTINGBOT_SECRET environment variables.

There are three ways to provide them to CodeBuild, in order of preference:

1. AWS Secrets Manager (recommended)

Store your credentials as a secret, for example a secret named testingbot with the keys key and secret, then reference it from your buildspec.yml:

env:
  secrets-manager:
    TESTINGBOT_KEY: testingbot:key
    TESTINGBOT_SECRET: testingbot:secret

The CodeBuild service role needs secretsmanager:GetSecretValue permission on that secret. The values never appear in your repository and CodeBuild masks them in the build log.

2. AWS Systems Manager Parameter Store

env:
  parameter-store:
    TESTINGBOT_KEY: /testingbot/key
    TESTINGBOT_SECRET: /testingbot/secret

Store both as SecureString parameters and grant the build role ssm:GetParameters.

3. Plain environment variables

You can also set the variables directly on the CodeBuild project, under Environment → Additional configuration → Environment variables, or in the build stage of the Create Pipeline wizard.

env:
  variables:
    TESTINGBOT_KEY: "YOUR_KEY"
    TESTINGBOT_SECRET: "YOUR_SECRET"

Create a Pipeline

In the AWS console, go to CodePipeline and choose Create pipeline:

1. Pipeline settings: give the pipeline a name and let AWS create a new service role.
2. Source stage: pick your source provider (GitHub, Bitbucket, CodeCommit or S3), the repository and the branch that should trigger the pipeline.
3. Build stage: choose AWS CodeBuild as the build provider and select your build project. It helps to create the CodeBuild project beforehand, because its name is what you will recognize in your build logs. Add the TestingBot environment variables here if you are not using Secrets Manager.
4. Deploy stage: optional, select a deploy provider if your pipeline also deploys.
5. Review the configuration and choose Create pipeline.

From now on, every commit to the configured branch triggers the pipeline and runs your tests on TestingBot.

Buildspec Example

Below is a buildspec.yml that installs the project dependencies and runs a test suite against the TestingBot cloud:

version: 0.2

env:
  secrets-manager:
    TESTINGBOT_KEY: testingbot:key
    TESTINGBOT_SECRET: testingbot:secret

phases:
  install:
    runtime-versions:
      nodejs: 20
    commands:
      - npm ci
  build:
    commands:
      - npm run test:e2e

reports:
  e2e:
    files:
      - "junit/*.xml"

The optional reports section makes CodeBuild display your JUnit results in the AWS console, next to the full test reports, videos and logs on TestingBot.

Locally Hosted Websites

If the website you want to test is not publicly reachable (a staging environment inside a VPC, or a server started by the build itself), start a TestingBot Tunnel in the pre_build phase. The tunnel is a Java application, so add a Java runtime to the build image.

version: 0.2

env:
  secrets-manager:
    TESTINGBOT_KEY: testingbot:key
    TESTINGBOT_SECRET: testingbot:secret

phases:
  install:
    runtime-versions:
      nodejs: 20
      java: corretto17
    commands:
      - npm ci
  pre_build:
    commands:
      - curl -sS -O https://testingbot.com/downloads/testingbot-tunnel.zip
      - unzip -o -q testingbot-tunnel.zip
      - |
        java -jar testingbot-tunnel-*.jar $TESTINGBOT_KEY $TESTINGBOT_SECRET \
          --tunnel-identifier codebuild-$CODEBUILD_BUILD_NUMBER \
          --readyfile /tmp/tunnel.ready \
          --logfile /tmp/tunnel.log &
      - timeout 180 bash -c 'until [ -f /tmp/tunnel.ready ]; do sleep 1; done'
  build:
    commands:
      - npm run start:staging &
      - npm run test:e2e
  post_build:
    commands:
      - pkill -f testingbot-tunnel || true

artifacts:
  files:
    - /tmp/tunnel.log

A few things worth pointing out:

• The --readyfile flag touches a file as soon as the tunnel is ready. Waiting for that file is more reliable than a fixed sleep, because your tests never start before the tunnel is up.
• The --tunnel-identifier flag tags the tunnel, so parallel builds each get their own tunnel. Pass the same value as the tunnel-identifier capability in your tests. CODEBUILD_BUILD_NUMBER is set by CodeBuild for every build.
• The post_build phase always runs, also when your tests fail, which makes it the right place to stop the tunnel.
• Uploading /tmp/tunnel.log as an artifact makes it easy to debug connection problems afterwards.

If your staging environment lives in a private VPC, configure the CodeBuild project to run inside that VPC. The tunnel then reaches your internal hosts the same way the build does.

Example Test

Your test reads the credentials from the environment and sends the TestingBot capabilities in the tb:options object:

#!/usr/bin/env ruby

require 'rubygems'
require 'selenium-webdriver'

options = Selenium::WebDriver::Chrome::Options.new
options.add_option('platformName', 'WIN11')
options.add_option('browserVersion', 'latest')
options.add_option('tb:options', {
  'key' => ENV['TESTINGBOT_KEY'],
  'secret' => ENV['TESTINGBOT_SECRET'],
  'name' => 'CodePipeline Test',
  'build' => "codebuild-#{ENV['CODEBUILD_BUILD_NUMBER']}",
  'tunnel-identifier' => "codebuild-#{ENV['CODEBUILD_BUILD_NUMBER']}"
})

driver = Selenium::WebDriver.for(
  :remote,
  url: "https://hub.testingbot.com/wd/hub",
  options: options)
driver.navigate.to "http://localhost:8080"
puts driver.title
driver.quit

import os
from selenium import webdriver
from selenium.webdriver.chrome.options import Options

build = "codebuild-%s" % os.environ.get('CODEBUILD_BUILD_NUMBER')

options = Options()
options.set_capability('platformName', 'WIN11')
options.set_capability('browserVersion', 'latest')
options.set_capability('tb:options', {
  'key': os.environ['TESTINGBOT_KEY'],
  'secret': os.environ['TESTINGBOT_SECRET'],
  'name': 'CodePipeline Test',
  'build': build,
  'tunnel-identifier': build
})

driver = webdriver.Remote(
    command_executor='https://hub.testingbot.com/wd/hub',
    options=options
)
driver.get("http://localhost:8080")
print(driver.title)
driver.quit()

import org.openqa.selenium.WebDriver;
import org.openqa.selenium.chrome.ChromeOptions;
import org.openqa.selenium.remote.RemoteWebDriver;

import java.net.URL;
import java.util.HashMap;
import java.util.Map;

public class CodePipelineTest {

  public static final String KEY = System.getenv("TESTINGBOT_KEY");
  public static final String SECRET = System.getenv("TESTINGBOT_SECRET");
  public static final String URL = "https://hub.testingbot.com/wd/hub";

  public static void main(String[] args) throws Exception {
    String build = "codebuild-" + System.getenv("CODEBUILD_BUILD_NUMBER");

    ChromeOptions options = new ChromeOptions();
    options.setCapability("browserVersion", "latest");
    options.setCapability("platformName", "WIN11");

    Map<String, Object> tbOptions = new HashMap<>();
    tbOptions.put("key", KEY);
    tbOptions.put("secret", SECRET);
    tbOptions.put("name", "CodePipeline Test");
    tbOptions.put("build", build);
    tbOptions.put("tunnel-identifier", build);
    options.setCapability("tb:options", tbOptions);

    WebDriver driver = new RemoteWebDriver(new URL(URL), options);
    driver.get("http://localhost:8080");

    System.out.println(driver.getTitle());

    driver.quit();
  }
}

const webdriver = require('selenium-webdriver');
const chrome = require('selenium-webdriver/chrome');

const build = `codebuild-${process.env.CODEBUILD_BUILD_NUMBER}`;

async function runCodePipelineTest () {
  let options = new chrome.Options();
  options.set('platformName', 'WIN11');
  options.set('browserVersion', 'latest');
  options.set('tb:options', {
    'key': process.env.TESTINGBOT_KEY,
    'secret': process.env.TESTINGBOT_SECRET,
    'name': 'CodePipeline Test',
    'build': build,
    'tunnel-identifier': build
  });

  let driver = new webdriver.Builder()
    .usingServer('https://hub.testingbot.com/wd/hub')
    .withCapabilities(options)
    .build();
  await driver.get("http://localhost:8080");
  const title = await driver.getTitle();
  console.log(title);
  await driver.quit();
}
runCodePipelineTest();

Leave out the tunnel-identifier capability when you are testing a public website without a tunnel.

Grouping Builds

By passing the build capability, all sessions started by one CodeBuild run are grouped into a single TestingBot build, which gives you one combined pass or fail result for the pipeline execution.

CodeBuild exposes several variables you can use for this:

You can report the outcome of your build back to TestingBot with the TestingBot REST API, or fetch the status of a build to gate a deploy stage on the test results.

Verify the Results

In the AWS console, open CodePipeline, select your pipeline and choose Details on the build stage to follow the CodeBuild log.

At the same time, your sessions appear in the TestingBot member area, with video, screenshots, WebDriver logs and network traffic for every test.

Troubleshooting

• Authentication failed: the build could not read your credentials. Check that the CodeBuild service role is allowed to read the secret or parameter, and that the key and secret are not swapped.
• The tunnel never becomes ready: make sure a Java runtime is present in the install phase (java: corretto17) and inspect /tmp/tunnel.log. If your VPC blocks outbound traffic, allow port 443 to hub.testingbot.com and *.testingbot.com.
• Tests time out on a local URL: verify that the tunnel-identifier capability matches the --tunnel-identifier value passed to the tunnel, and that your local server is already listening when the tests start.
• The build hangs after the tests: the tunnel is still running in the background. Stop it in the post_build phase as shown above.