Features

Azure Devops (VSTS)

Azure Devops (formerly Team Foundation Server and Visual Studio Team System) is a CI service owned by Microsoft.

TestingBot has a Marketplace Extension that offers some features to better integrate TestingBot with Azure Devops:

  • Set TestingBot environment variables (TB_KEY and TB_SECRET) to be used by your tests.
  • Embed overview of your tests in Azure Devops.
  • Start/Stop TestingBot Tunnel for specific builds.

Installing Azure Pipelines extension

  1. Make sure you are logged in to Azure Devops.
    Then please go to the Visual Studio Marketplace and download the TestingBot extension.

    TestingBot Azure extension
  2. Click on Get it free to install the extension.

  3. Using the Select an Azure DevOps organization drop-down menu, choose the organization and click install.

    install the TestingBot Azure extension

Configure the pipeline

Now that the TestingBot extension has been installed, you'll need to configure the Service Connection.

  • Go to Project Settings and select Service connections.

    project settings
  • Click New Service Connection.
    You can now add the TestingBot credentials. service connection new service connection

    • Your TestingBot API Key as Username: see the TestingBot member dashboard

    • Your TestingBot API Secret as Password: see the TestingBot member dashboard

    new service credentials

    You can leave the Server URL to the default value: https://api.testingbot.com/v1/

  • Add a Service connection name and click Save.

Now that we've set up the Service Connection, we can go ahead and add the custom TestingBot task to the azure-pipelines.yml file.
To do this, you can edit the file in your favorite editor, or through the YAML editor on Azure Devops.

TestingBot Task

In the steps block, you can now add the TBMain task, either via the Task menu in the Azure Devops page or by editing the file:

Copy code
- task: TBMain@0
  inputs:
    connectedServiceName: '{your service-connection-name}'
TestingBot task TestingBot credentials

Viewing Test Results

The extension allows you to view test results from Azure.
To enable this, make sure you followed the previous steps and modify your existing test or test suite to send an extra capability to TestingBot: build.

The extension makes the following environment variables available for your test cases:

  • TB_KEY
  • TB_SECRET
  • TB_BUILD_NAME

The TB_BUILD_NAME environment variable needs to be passed as a build capability, for example:

Copy code
DesiredCapabilities capabilities = new DesiredCapabilities();

capabilities.setCapability(CapabilityType.BROWSER_NAME, browser);
capabilities.setCapability(CapabilityType.VERSION, version);
capabilities.setCapability(CapabilityType.PLATFORM, os);
capabilities.setCapability("name", methodName);
// this capability below will make sure the reporting works
capabilities.setCapability("build", System.getenv("TB_BUILD_NAME"));

webDriver.set(new RemoteWebDriver(
  new URL("https://" + System.getenv("TB_KEY") + ":" + System.getenv("TB_SECRET") + "@hub.testingbot.com/wd/hub"),
capabilities));

It is important to pass the TB_BUILD_NAME environment variable as a build capability for the reporting to work.

Once the build is complete, you will see a TestingBot tab. This will show an overview of the tests for this build.

TestingBot embed test results

Enable TestingBot Tunnel

TestingBot Tunnel allows you to run tests against web applications hosted on private, staging or development machines.
To enable the TestingBot Tunnel in Azure, make sure the TestingBot Tunnel checkbox is checked.

TestingBot enable tunnel

To use the Tunnel with your test, make sure to edit your test to use the Tunnel IP (localhost:4445):

Copy code
DesiredCapabilities capabilities = new DesiredCapabilities();

capabilities.setCapability(CapabilityType.BROWSER_NAME, browser);
capabilities.setCapability(CapabilityType.VERSION, version);
capabilities.setCapability(CapabilityType.PLATFORM, os);
capabilities.setCapability("name", methodName);
// this capability below will make sure the reporting works
capabilities.setCapability("build", System.getenv("TB_BUILD_NAME"));

webDriver.set(new RemoteWebDriver(
  new URL("https://" + System.getenv("TB_KEY") + ":" + System.getenv("TB_SECRET") + "@localhost:4445/wd/hub"),
capabilities));

Stop Tunnel

It's important to close the tunnel at the end of your test. To do this, please add the Stop TestingBot Tunnel task in your azure-pipelines.yml file.

Make sure to add the TBStopTunnel@0 after your other tasks.

TestingBot stop tunnel