Features
Start a free trial

TestNG & Cucumber Automated Testing

Please consult the TestNG + Cucumber example repository to see a simple example on how to run TestNG & Cucumber tests on TestingBot.

TestNG is a powerful testing framework inspired by JUnit and NUnit but with more flexible and robust features. It is widely used by Java enthusiast for testing because it offers powerful annotations, test configuration and parallel testing.

Cucumber is a popular tool for Behavior-Driven Development (BDD), allowing test scripts to be written in a human-readable format using the Gherkin language. Because Cucumber uses this natural-like language, it enables collaboration between technical and non-technical team members.

To get started, let's go through all the necessary steps required to run your first test with TestNG and Cucumber on TestingBot.

Prerequisites

Let's start with making sure Java is available on your system.

For Windows:
  • You can download Java for Windows from Java.com
  • Run the installer and follow the setup wizard to install Java.
For Linux:
Copy code
sudo apt-get install openjdk-11-jre
For macOS:

Java should already be present on macOS by default.

Set up pom.xml dependencies

Now we can specify which dependencies we need to be able to run a test. Create a new file called pom.xml and make sure to define at least the following dependencies:

  • io.cucumber:cucumber-java io.cucumber:cucumber-testng org.testng:testng org.seleniumhq.selenium:selenium-java
Copy code
<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>
    <artifactId>testingbot-testng-cucumber</artifactId>
    <groupId>com.testingbot</groupId>
    <version>1.0-SNAPSHOT</version>
    <packaging>jar</packaging>
    <name>testingbot_cucumber_testng</name>
    <description>An example project to run Cucumber + TestNG tests on TestingBot's browser grid</description>
    <dependencies>
        <dependency>
            <groupId>io.cucumber</groupId>
            <artifactId>cucumber-java</artifactId>
            <version>7.18.1</version>
        </dependency>
        <dependency>
            <groupId>io.cucumber</groupId>
            <artifactId>cucumber-testng</artifactId>
            <version>7.18.1</version>
        </dependency>
        <dependency>
            <groupId>org.testng</groupId>
            <artifactId>testng</artifactId>
            <version>7.10.2</version>
            <scope>test</scope>
        </dependency>
        <dependency>
            <groupId>org.seleniumhq.selenium</groupId>
            <artifactId>selenium-java</artifactId>
            <version>4.24.0</version>
            <scope>test</scope>
        </dependency>
    </dependencies>
    <build>
        <plugins>
            <plugin>
                <artifactId>maven-compiler-plugin</artifactId>
                <version>3.0</version>
                <configuration>
                    <source>1.7</source>
                    <target>1.7</target>
                </configuration>
            </plugin>
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-surefire-plugin</artifactId>
                <version>2.12.4</version>
                <configuration>
                	<parallel>classes</parallel>
          			<threadCount>40</threadCount>
                    <redirectTestOutputToFile>false</redirectTestOutputToFile>
                </configuration>
            </plugin>
        </plugins>
    </build>
</project>

Project Structure

Now we need to create a folder structure that will contain the .feature files and step definitions.

Create a folder for the .feature files:

Copy code
mkdir -p src/test/resources/features

We can also define directories for the step definitions, driver and the TestingBot runner class:

Copy code
mkdir -p src/test/java/com/testingbotdemo/selenium/cucumber/{definitions,drivers,runners}

Define TestingBot driver

Now we can define a custom driver that will run the tests on the TestingBot grid:

Copy code
package com.testingbotdemo.selenium.cucumber.drivers;

import org.openqa.selenium.WebDriver;
import org.openqa.selenium.remote.DesiredCapabilities;
import org.openqa.selenium.remote.RemoteWebDriver;
import java.net.URL;
import java.net.MalformedURLException;
import java.util.HashMap;
import java.util.Map;

public class TestingBotDriver {

    public static WebDriver getDriver() {
        String username = System.getenv("TESTINGBOT_KEY") == null ? "TESTINGBOT_KEY" : System.getenv("TESTINGBOT_KEY");
        String accesskey = System.getenv("TESTINGBOT_SECRET") == null ? "TESTINGBOT_SECRET" : System.getenv("TESTINGBOT_SECRET");

        DesiredCapabilities capabilities = new DesiredCapabilities();
        capabilities.setCapability("browserName", "chrome");
        capabilities.setCapability("browserVersion", "latest");
        capabilities.setCapability("platformName", "WIN10");

        Map<String, Object> testingBotOptions = new HashMap<>();
        testingBotOptions.put("name", "Cucumber Example");
        capabilities.setCapability("tb:options", testingBotOptions);

        try {
            String gridURL = "https://" + username + ":" + accesskey + "@hub.testingbot.com/wd/hub";
            return new RemoteWebDriver(new URL(gridURL), capabilities);
        } catch (MalformedURLException e) {
            e.printStackTrace();
            throw new RuntimeException("Invalid URL provided for TestingBot hub", e);
        }
    }
}

We will need to make sure to later specify the TESTINGBOT_KEY and TESTINGBOT_SECRET environment variables to authenticate and run tests on the TestingBot remote browser grid.

The DesiredCapabilities are used to define which remote browser we would like to run this Cucumber test on.

Test Runner

Next, let's create a Test runner class that will use the CucumberOptions annotation to let Cucumber know where to find the feature files and the step definitions.

Copy code
package com.testingbotdemo.selenium.cucumber.runners;

import io.cucumber.testng.AbstractTestNGCucumberTests;
import io.cucumber.testng.CucumberOptions;

@CucumberOptions(
        features = "src/test/resources/features",
        glue = "com.testingbotdemo.selenium.cucumber.definitions"
)
public class TestRunner extends AbstractTestNGCucumberTests {
}

Step Definitions

Finally, we need to define the step definitions. This is the logic that converts the Gherkin syntax to actual code.

The code that we will use will interact with the Driver class we defined.

Copy code
package com.testingbotdemo.selenium.cucumber.definitions;

import io.cucumber.java.en.Given;
import io.cucumber.java.en.Then;
import io.cucumber.java.en.When;
import org.openqa.selenium.By;
import org.testng.Assert;
import com.testingbotdemo.selenium.cucumber.drivers.TestingBotDriver;
import org.openqa.selenium.WebDriver;

public class StepDefinitions {
    WebDriver driver = TestingBotDriver.getDriver();

    @Given("I open Google")
    public void i_open_google() {
        driver.get("https://www.google.com");
    }

    @When("I search for {string}")
    public void i_search_for(String query) {
        driver.findElement(By.name("q")).sendKeys(query);
        driver.findElement(By.name("btnK")).submit();
    }

    @Then("I should see results")
    public void i_should_see_results() {
        Assert.assertTrue(driver.getTitle().contains("Google"));
        driver.quit();
    }
}

Run Cucumber test

You can now run the Cucumber test:

Copy code
TB_KEY={..your-key..} TB_SECRET={..your-secret-..} mvn test

The test will run on a remote TestingBot browser. Once the test has finished, you will be able to see the result, together with a video and logs in the TestingBot dashboard.

Specify Browsers & Devices

To let TestingBot know on which browser/platform/device you want to run your test on, you need to specify the browsername, version, OS and other optional options in the capabilities field.

1. Select a Platform
2. Select a browser
Copy code

Testing Internal Websites

We've built TestingBot Tunnel, to provide you with a secure way to run tests against your staged/internal webapps.
Please see our TestingBot Tunnel documentation for more information about this easy to use tunneling solution.

The example below shows how to easily run a TestNG test with our Tunnel:

1. Download our tunnel and start the tunnel:

Copy code
java -jar testingbot-tunnel.jar key secret

2. Adjust your test: instead of pointing to 'hub.testingbot.com/wd/hub' like the example above - change it to point to your tunnel's IP address.
Assuming you run the tunnel on the same machine you run your tests, change to 'localhost:4445/wd/hub'. localhost is the machine running the tunnel, 4445 is the default port of the tunnel.

This way your test will go securely through the tunnel to TestingBot and back:

Copy code
package com.testingbotdemo.selenium.cucumber.drivers;

import org.openqa.selenium.WebDriver;
import org.openqa.selenium.remote.DesiredCapabilities;
import org.openqa.selenium.remote.RemoteWebDriver;
import java.net.URL;
import java.net.MalformedURLException;
import java.util.HashMap;
import java.util.Map;

public class TestingBotDriver {

    public static WebDriver getDriver() {
        String username = System.getenv("TESTINGBOT_KEY") == null ? "TESTINGBOT_KEY" : System.getenv("TESTINGBOT_KEY");
        String accesskey = System.getenv("TESTINGBOT_SECRET") == null ? "TESTINGBOT_SECRET" : System.getenv("TESTINGBOT_SECRET");

        DesiredCapabilities capabilities = new DesiredCapabilities();
        capabilities.setCapability("browserName", "chrome");
        capabilities.setCapability("browserVersion", "latest");
        capabilities.setCapability("platformName", "WIN10");

        Map<String, Object> testingBotOptions = new HashMap<>();
        testingBotOptions.put("name", "Cucumber Example");
        capabilities.setCapability("tb:options", testingBotOptions);

        try {
            String gridURL = "http://" + username + ":" + accesskey + "@localhost:4445/wd/hub";
            return new RemoteWebDriver(new URL(gridURL), capabilities);
        } catch (MalformedURLException e) {
            e.printStackTrace();
            throw new RuntimeException("Invalid URL provided for TestingBot hub", e);
        }
    }
}

Run tests in Parallel

Parallel Testing means running the same test, or multiple tests, simultaneously. This greatly reduces your total testing time.

You can run the same tests on all different browser configurations or run different tests all on the same browser configuration.
TestingBot has a large grid of machines and browsers, which means you can use our service to do efficient parallel testing. It is one of the key features we provide to greatly cut down on your total testing time.

To run a test on different browsers at the same time, you can run the same mvn test command multiple times in parallel, each with a different browser/capability environment variable.

For example, assuming you define the desired capabilities with environment variables:

Copy code
DesiredCapabilities capabilities = new DesiredCapabilities();
capabilities.setCapability("browserName", System.getenv("CAPABILITY_BROWSERNAME"));
capabilities.setCapability("browserVersion", System.getenv("CAPABILITY_BROWSERVERSION"));
capabilities.setCapability("platformName", System.getenv("CAPABILITY_PLATFORMNAME"));

Now you can run the same test on multiple browsers simultaneously:

Copy code
mvn test -DCAPABILITY_BROWSERNAME=chrome -DCAPABILITY_BROWSERVERSION=latest -DCAPABILITY_PLATFORMNAME=WIN10
mvn test -DCAPABILITY_BROWSERNAME=safari -DCAPABILITY_BROWSERVERSION=latest -DCAPABILITY_PLATFORMNAME=SONOMA

Queuing

Every plan we provide comes with a limit of parallel tests.
If you exceed the number of parallel tests assigned to your account, TestingBot will queue the additional tests (for up to 6 minutes) and run the tests as soon as slots become available.

Mark tests as passed/failed

To see if a test passed or not in our member area, or to send additional meta-data to TestingBot, you can use the TestingBot API.

TestingBot has a Java client to facilitate interacting with the TestingBot API.

Once you've included this client with your tests, you will be able to send back the test status and other meta-data to TestingBot:

Copy code
import com.testingbot.testingbotrest.TestingbotREST;

@After
public void tearDown() throws Exception {
  TestingbotREST r = new TestingbotREST("key", "secret");
  Map<String, String> data = new HashMap<String, String>();
  data.put("success", "1");
  data.put("name", "My Test");
  r.updateTest(driver.getSessionId(), data);
  driver.quit();
}

Other Java Framework examples

  • JUnit

    JUnit is a unit testing framework for the Java programming language.

  • Parallel JUnit

    By running multiple JUnit tests at the same time you can cut down on overall test time.

  • TestNG

    TestNG is a framework similar to JUnit and NUnit, which supports some additional commands and features.