AWS CodePipeline Automated Testing

Prerequisites

• An AWS account with permission to create CodePipeline and CodeBuild projects.
• A repository with your tests, connected to CodePipeline as a source.
• A buildspec.yml file in the root of that repository.
• A TestingBot key and secret, available in the TestingBot member area.

Credentials

Your tests read the TestingBot key and secret from the TESTINGBOT_KEY and TESTINGBOT_SECRET environment variables.

There are three ways to provide them to CodeBuild, in order of preference:

1. AWS Secrets Manager (recommended)

Store your credentials as a secret, for example a secret named testingbot with the keys key and secret, then reference it from your buildspec.yml:

env:
  secrets-manager:
    TESTINGBOT_KEY: testingbot:key
    TESTINGBOT_SECRET: testingbot:secret

The CodeBuild service role needs secretsmanager:GetSecretValue permission on that secret. The values never appear in your repository and CodeBuild masks them in the build log.

2. AWS Systems Manager Parameter Store

env:
  parameter-store:
    TESTINGBOT_KEY: /testingbot/key
    TESTINGBOT_SECRET: /testingbot/secret

Store both as SecureString parameters and grant the build role ssm:GetParameters.

3. Plain environment variables

You can also set the variables directly on the CodeBuild project, under Environment → Additional configuration → Environment variables, or in the build stage of the Create Pipeline wizard.

env:
  variables:
    TESTINGBOT_KEY: "YOUR_KEY"
    TESTINGBOT_SECRET: "YOUR_SECRET"

TestingBot for CodeBuild

testingbot-codepipeline is a small Python package that installs the testingbot-codebuild command. It has no dependencies and needs Python 3.9 or newer, which the standard CodeBuild Linux images (x64 and arm64) include.

version: 0.2

env:
  secrets-manager:
    TESTINGBOT_KEY: testingbot:key
    TESTINGBOT_SECRET: testingbot:secret
  exported-variables:
    - TESTINGBOT_BUILD

phases:
  install:
    commands:
      - pip install --quiet testingbot-codepipeline
      - npm ci
  pre_build:
    commands:
      - eval "$(testingbot-codebuild env)"
      - testingbot-codebuild tunnel start
  build:
    commands:
      - npm run test:e2e
  post_build:
    commands:
      - testingbot-codebuild tunnel stop
      - testingbot-codebuild report

reports:
  testingbot:
    files:
      - testingbot-reports/testingbot.xml
    file-format: JUNITXML

What each testingbot-codebuild command does:

Your tests pass the two variables as the build and tunnel-identifier capabilities:

'tb:options': {
  'key': process.env.TESTINGBOT_KEY,
  'secret': process.env.TESTINGBOT_SECRET,
  'build': process.env.TESTINGBOT_BUILD,
  'tunnel-identifier': process.env.TESTINGBOT_TUNNEL_IDENTIFIER
}

The tunnel receives your key and secret through its environment, so they never appear on a command line or in the build log. To keep the tunnel download between builds, add /root/.testingbot-tunnel/**/* to the buildspec cache paths.

CodeBuild Test Reports

testingbot-codebuild report fetches every test of the TestingBot build and writes testingbot-reports/testingbot.xml. The reports section of the buildspec above hands that file to CodeBuild, which shows the results on the build's Reports tab:

• one test suite per browser and operating system, so you see where a test failed without opening it;
• the failure message you reported to TestingBot, and a link to the test with its video, screenshots and logs;
• tests that never reported a pass or fail status to TestingBot show as skipped, not failed. Set the status from your test with the TestingBot REST API or your framework's TestingBot integration.

The CodeBuild service role needs codebuild:CreateReportGroup, codebuild:CreateReport, codebuild:UpdateReport and codebuild:BatchPutTestCases for the report group.

Deploy Gate

The TestingBot deploy gate is an AWS Lambda function for a CodePipeline Invoke action. Place it after the action that runs your tests and before your deploy stage. It passes the pipeline when the TestingBot build passed, and otherwise fails it with the name and link of every failing test.

1. Deploy the gate from the AWS Serverless Application Repository. It asks for TestingBotSecretArn, the ARN of the Secrets Manager secret holding your key and secret. The gate creates its own IAM role, so tick I acknowledge that this app creates custom IAM roles.
2. On the CodeBuild action that runs your tests, set the variable namespace to TestVariables, and keep TESTINGBOT_BUILD under exported-variables in the buildspec.
3. Add an action after it with action provider AWS Lambda, the gate's function name, and these user parameters:

   {"build": "#{TestVariables.TESTINGBOT_BUILD}"}

4. Allow the pipeline's service role lambda:InvokeFunction on the gate.

The user parameters accept these optional settings:

A complete CloudFormation example of the test and gate actions is in the repository. You can also deploy the gate from source with sam deploy --guided.

Create a Pipeline

In the AWS console, go to CodePipeline and choose Create pipeline:

1. Pipeline settings: give the pipeline a name and let AWS create a new service role.
2. Source stage: pick your source provider (GitHub, Bitbucket, CodeCommit or S3), the repository and the branch that should trigger the pipeline.
3. Build stage: choose AWS CodeBuild as the build provider and select your build project. It helps to create the CodeBuild project beforehand, because its name is what you will recognize in your build logs. Add the TestingBot environment variables here if you are not using Secrets Manager.
4. Deploy stage: optional, select a deploy provider if your pipeline also deploys.
5. Review the configuration and choose Create pipeline.

From now on, every commit to the configured branch triggers the pipeline and runs your tests on TestingBot.

Manual Buildspec

If you prefer not to install testingbot-codebuild, the rest of this page shows how to do the same steps by hand.

Below is a buildspec.yml that installs the project dependencies and runs a test suite against the TestingBot cloud:

version: 0.2

env:
  secrets-manager:
    TESTINGBOT_KEY: testingbot:key
    TESTINGBOT_SECRET: testingbot:secret

phases:
  install:
    runtime-versions:
      nodejs: 20
    commands:
      - npm ci
  build:
    commands:
      - npm run test:e2e

reports:
  e2e:
    files:
      - "junit/*.xml"

The optional reports section makes CodeBuild display your JUnit results in the AWS console, next to the full test reports, videos and logs on TestingBot.

Locally Hosted Websites

If the website you want to test is not publicly reachable (a staging environment inside a VPC, or a server started by the build itself), start a TestingBot Tunnel in the pre_build phase. The tunnel is a Java application, so add a Java runtime to the build image.

version: 0.2

env:
  secrets-manager:
    TESTINGBOT_KEY: testingbot:key
    TESTINGBOT_SECRET: testingbot:secret

phases:
  install:
    runtime-versions:
      nodejs: 20
      java: corretto17
    commands:
      - npm ci
  pre_build:
    commands:
      - curl -sS -O https://testingbot.com/downloads/testingbot-tunnel.zip
      - unzip -o -q testingbot-tunnel.zip
      - |
        java -jar testingbot-tunnel.jar $TESTINGBOT_KEY $TESTINGBOT_SECRET \
          --tunnel-identifier codebuild-$CODEBUILD_BUILD_NUMBER \
          --readyfile /tmp/tunnel.ready \
          --logfile /tmp/tunnel.log &
      - timeout 180 bash -c 'until [ -f /tmp/tunnel.ready ]; do sleep 1; done'
  build:
    commands:
      - npm run start:staging &
      - npm run test:e2e
  post_build:
    commands:
      - pkill -f testingbot-tunnel || true

artifacts:
  files:
    - /tmp/tunnel.log

A few things worth pointing out:

• The --readyfile flag touches a file as soon as the tunnel is ready. Waiting for that file is more reliable than a fixed sleep, because your tests never start before the tunnel is up.
• The --tunnel-identifier flag tags the tunnel, so parallel builds each get their own tunnel. Pass the same value as the tunnel-identifier capability in your tests. CODEBUILD_BUILD_NUMBER is set by CodeBuild for every build.
• The post_build phase always runs, also when your tests fail, which makes it the right place to stop the tunnel.
• Uploading /tmp/tunnel.log as an artifact makes it easy to debug connection problems afterwards.

If your staging environment lives in a private VPC, configure the CodeBuild project to run inside that VPC. The tunnel then reaches your internal hosts the same way the build does.

Example Test

Your test reads the credentials from the environment and sends the TestingBot capabilities in the tb:options object:

#!/usr/bin/env ruby

require 'rubygems'
require 'selenium-webdriver'

options = Selenium::WebDriver::Chrome::Options.new
options.platform_name = 'WIN11'
options.browser_version = 'latest'
options.add_option('tb:options', {
  'key' => ENV['TESTINGBOT_KEY'],
  'secret' => ENV['TESTINGBOT_SECRET'],
  'name' => 'CodePipeline Test',
  'build' => "codebuild-#{ENV['CODEBUILD_BUILD_NUMBER']}",
  'tunnel-identifier' => "codebuild-#{ENV['CODEBUILD_BUILD_NUMBER']}"
})

driver = Selenium::WebDriver.for(
  :remote,
  url: "https://hub.testingbot.com/wd/hub",
  options: options)
driver.navigate.to "http://localhost:8080"
puts driver.title
driver.quit

import os
from selenium import webdriver
from selenium.webdriver.chrome.options import Options

build = "codebuild-%s" % os.environ.get('CODEBUILD_BUILD_NUMBER')

options = Options()
options.set_capability('platformName', 'WIN11')
options.set_capability('browserVersion', 'latest')
options.set_capability('tb:options', {
  'key': os.environ['TESTINGBOT_KEY'],
  'secret': os.environ['TESTINGBOT_SECRET'],
  'name': 'CodePipeline Test',
  'build': build,
  'tunnel-identifier': build
})

driver = webdriver.Remote(
    command_executor='https://hub.testingbot.com/wd/hub',
    options=options
)
driver.get("http://localhost:8080")
print(driver.title)
driver.quit()

import org.openqa.selenium.WebDriver;
import org.openqa.selenium.chrome.ChromeOptions;
import org.openqa.selenium.remote.RemoteWebDriver;

import java.net.URL;
import java.util.HashMap;
import java.util.Map;

public class CodePipelineTest {

  public static final String KEY = System.getenv("TESTINGBOT_KEY");
  public static final String SECRET = System.getenv("TESTINGBOT_SECRET");
  public static final String URL = "https://hub.testingbot.com/wd/hub";

  public static void main(String[] args) throws Exception {
    String build = "codebuild-" + System.getenv("CODEBUILD_BUILD_NUMBER");

    ChromeOptions options = new ChromeOptions();
    options.setCapability("browserVersion", "latest");
    options.setCapability("platformName", "WIN11");

    Map<String, Object> tbOptions = new HashMap<>();
    tbOptions.put("key", KEY);
    tbOptions.put("secret", SECRET);
    tbOptions.put("name", "CodePipeline Test");
    tbOptions.put("build", build);
    tbOptions.put("tunnel-identifier", build);
    options.setCapability("tb:options", tbOptions);

    WebDriver driver = new RemoteWebDriver(new URL(URL), options);
    driver.get("http://localhost:8080");

    System.out.println(driver.getTitle());

    driver.quit();
  }
}

const webdriver = require('selenium-webdriver');
const chrome = require('selenium-webdriver/chrome');

const build = `codebuild-${process.env.CODEBUILD_BUILD_NUMBER}`;

async function runCodePipelineTest () {
  let options = new chrome.Options();
  options.set('platformName', 'WIN11');
  options.set('browserVersion', 'latest');
  options.set('tb:options', {
    'key': process.env.TESTINGBOT_KEY,
    'secret': process.env.TESTINGBOT_SECRET,
    'name': 'CodePipeline Test',
    'build': build,
    'tunnel-identifier': build
  });

  let driver = new webdriver.Builder()
    .usingServer('https://hub.testingbot.com/wd/hub')
    .withCapabilities(options)
    .build();
  await driver.get("http://localhost:8080");
  const title = await driver.getTitle();
  console.log(title);
  await driver.quit();
}
runCodePipelineTest();

Leave out the tunnel-identifier capability when you are testing a public website without a tunnel.

Grouping Builds

By passing the build capability, all sessions started by one CodeBuild run are grouped into a single TestingBot build, which gives you one combined pass or fail result for the pipeline execution.

CodeBuild exposes several variables you can use for this:

To stop a deploy when TestingBot tests fail, use the deploy gate, which reads this build.

Verify the Results

In the AWS console, open CodePipeline, select your pipeline and choose Details on the build stage to follow the CodeBuild log.

At the same time, your sessions appear in the TestingBot member area, with video, screenshots, WebDriver logs and network traffic for every test.

Troubleshooting

• The gate says the build variable was not resolved: the variable namespace on the test action does not match the one in the user parameters, or TESTINGBOT_BUILD is missing from exported-variables.
• The gate or the report finds no tests: your tests do not send the build capability, or send a different value than $TESTINGBOT_BUILD.
• Authentication failed: the build could not read your credentials. Check that the CodeBuild service role is allowed to read the secret or parameter, and that the key and secret are not swapped.
• The tunnel never becomes ready: make sure a Java runtime is present in the install phase (java: corretto17) and inspect /tmp/tunnel.log. If your VPC blocks outbound traffic, allow port 443 to hub.testingbot.com and *.testingbot.com.
• Tests time out on a local URL: verify that the tunnel-identifier capability matches the --tunnel-identifier value passed to the tunnel, and that your local server is already listening when the tests start.
• The build hangs after the tests: the tunnel is still running in the background. Stop it in the post_build phase as shown above.