Exit course
Chapter 10 of 11 Intermediate 8 min read 3 quick checks Updated

Run the suite on a Selenium grid

On this page
  1. What changes and what doesn't
  2. Capabilities
  3. A driver factory
  4. The final BaseTest
  5. A cross-browser suite file
  6. Parallel limits and queueing
  7. Testing a site that is not public
  8. Debugging a remote failure

So far every browser has run on your own machine. That covers one operating system and whichever browser versions you happen to have installed, and parallelism is limited by your CPU. A Selenium grid runs the browsers on remote machines instead: your test code stays the same, but new ChromeDriver() becomes new RemoteWebDriver(gridUrl, options). This chapter makes that switch with a system property, using the TestingBot grid as the example. The same RemoteWebDriver code works with a self-hosted Selenium Grid; only the URL and the vendor-specific options change.

What changes and what doesn't#

Stays the same Changes
Test classes, locators, waits, assertions How the driver is created
testng.xml structure, groups, data providers The browser and OS now come from capabilities
ThreadLocal base class The thread count is bounded by the grid's parallel limit
Screenshot listener You also get a video and logs of each session

Because chapter 7 put driver creation in one place, the switch touches one class.

Capabilities#

A remote session is requested with the same ChromeOptions, FirefoxOptions or SafariOptions classes you use locally, plus three standard W3C capabilities:

  • browserName is set by the options class itself.
  • browserVersion, for example "latest" or a specific major version.
  • platformName, the operating system. On TestingBot these are names such as WIN11 for Windows 11 or SONOMA for macOS Sonoma.

Vendor-specific settings go in a namespaced map. TestingBot's is tb:options, where name sets the test name shown in the dashboard and build groups the tests of one run together. The capabilities generator builds the exact Java for any browser and OS combination, and test options lists everything else tb:options accepts, such as screen resolution, time zone and turning video recording off (it is on by default).

Where do you put the build name so every session of one CI run is grouped together in the TestingBot dashboard?

Vendor settings such as name and build go in tb:options. The suite name never reaches the grid.

A driver factory#

Move driver creation out of BaseTest into its own class, with a local and a remote branch:

package com.example;

import java.net.MalformedURLException;
import java.net.URI;
import java.util.HashMap;
import java.util.Map;
import org.openqa.selenium.MutableCapabilities;
import org.openqa.selenium.WebDriver;
import org.openqa.selenium.chrome.ChromeDriver;
import org.openqa.selenium.chrome.ChromeOptions;
import org.openqa.selenium.firefox.FirefoxDriver;
import org.openqa.selenium.firefox.FirefoxOptions;
import org.openqa.selenium.remote.RemoteWebDriver;
import org.openqa.selenium.safari.SafariOptions;

public final class DriverFactory {

    private DriverFactory() {}

    public static boolean onTestingBot() {
        return "testingbot".equals(System.getProperty("grid"));
    }

    public static WebDriver create(String browser, String platform, String testName)
            throws MalformedURLException {
        return onTestingBot() ? remote(browser, platform, testName) : local(browser);
    }

    private static WebDriver local(String browser) {
        boolean headless = Boolean.getBoolean("headless");
        if ("firefox".equals(browser)) {
            FirefoxOptions options = new FirefoxOptions();
            if (headless) options.addArguments("-headless");
            return new FirefoxDriver(options);
        }
        ChromeOptions options = new ChromeOptions();
        if (headless) options.addArguments("--headless=new", "--window-size=1920,1080");
        return new ChromeDriver(options);
    }

    private static WebDriver remote(String browser, String platform, String testName)
            throws MalformedURLException {
        MutableCapabilities options = switch (browser) {
            case "firefox" -> new FirefoxOptions();
            case "safari" -> new SafariOptions();
            default -> new ChromeOptions();
        };
        options.setCapability("platformName", platform);
        options.setCapability("browserVersion", "latest");

        Map<String, Object> tbOptions = new HashMap<>();
        tbOptions.put("name", testName);
        tbOptions.put("build", System.getProperty("build", "local"));
        options.setCapability("tb:options", tbOptions);

        String hub = "https://" + System.getenv("TB_KEY") + ":" + System.getenv("TB_SECRET")
                + "@hub.testingbot.com/wd/hub";
        return new RemoteWebDriver(URI.create(hub).toURL(), options);
    }
}

Credentials come from the TB_KEY and TB_SECRET environment variables, never from code or testng.xml. In CI, store them as secrets.

The final BaseTest#

BaseTest now names each remote session after the test method, and reports the result back before quitting:

package com.example;

import java.lang.reflect.Method;
import java.net.MalformedURLException;
import java.time.Duration;
import org.openqa.selenium.JavascriptExecutor;
import org.openqa.selenium.WebDriver;
import org.openqa.selenium.support.ui.WebDriverWait;
import org.testng.ITestResult;
import org.testng.annotations.AfterMethod;
import org.testng.annotations.BeforeMethod;
import org.testng.annotations.Optional;
import org.testng.annotations.Parameters;

public abstract class BaseTest {

    private static final ThreadLocal<WebDriver> DRIVER = new ThreadLocal<>();

    public static WebDriver driver() {
        return DRIVER.get();
    }

    protected WebDriverWait waitUpTo(int seconds) {
        return new WebDriverWait(driver(), Duration.ofSeconds(seconds));
    }

    @BeforeMethod(alwaysRun = true)
    @Parameters({"browser", "platform"})
    public void startDriver(@Optional("chrome") String browser,
                            @Optional("WIN11") String platform,
                            Method method) throws MalformedURLException {
        String testName = getClass().getSimpleName() + "." + method.getName();
        DRIVER.set(DriverFactory.create(browser, platform, testName));
    }

    @AfterMethod(alwaysRun = true)
    public void stopDriver(ITestResult result) {
        WebDriver driver = DRIVER.get();
        if (driver == null) {
            return;
        }
        try {
            if (DriverFactory.onTestingBot()) {
                String status = result.isSuccess() ? "passed" : "failed";
                ((JavascriptExecutor) driver).executeScript("tb:test-result=" + status);
            }
        } finally {
            driver.quit();
            DRIVER.remove();
        }
    }
}

@Parameters and the injected Method can be combined: TestNG fills the annotated parameters from testng.xml and injects Method by type.

Why report pass or fail?#

The grid only sees WebDriver commands. It cannot know that your Assert.assertEquals failed; from its point of view the session ended normally. Telling it the outcome makes the dashboard and build overview match your TestNG report. TestingBot supports two ways, both documented with examples:

  • Inside the session, via the JavaScript executor, as above: tb:test-result=passed or tb:test-result=failed. The same mechanism accepts tb:test-name=..., tb:test-build=... and tb:test-tags=.... It needs no extra dependency, but must run before quit().
  • After the session, via the REST API and the com.testingbot:testingbotrest Java client, calling updateTest(sessionId, data) with a success field (the docs use "1" for a passed test). Use this when you want to send more metadata or report from outside the test JVM.

A cross-browser suite file#

Create testng-grid.xml:

<!DOCTYPE suite SYSTEM "https://testng.org/testng-1.0.dtd">
<suite name="Grid" parallel="tests" thread-count="3">
  <listeners>
    <listener class-name="com.example.ScreenshotListener"/>
  </listeners>

  <test name="Chrome on Windows 11">
    <parameter name="browser" value="chrome"/>
    <parameter name="platform" value="WIN11"/>
    <packages><package name="com.example"/></packages>
  </test>

  <test name="Firefox on macOS Sonoma">
    <parameter name="browser" value="firefox"/>
    <parameter name="platform" value="SONOMA"/>
    <packages><package name="com.example"/></packages>
  </test>

  <test name="Safari on macOS Sonoma">
    <parameter name="browser" value="safari"/>
    <parameter name="platform" value="SONOMA"/>
    <packages><package name="com.example"/></packages>
  </test>
</suite>

Run it:

export TB_KEY=your-key
export TB_SECRET=your-secret
mvn test -Dgrid=testingbot -DsuiteXmlFile=testng-grid.xml -Dbuild="course-$(date +%Y%m%d-%H%M)"

parallel="tests" runs the three browsers side by side. To also parallelise inside each browser, switch to parallel="methods" and raise thread-count. The ThreadLocal base class handles both.

In CI, pass the pipeline's run number as -Dbuild=... so every test of one pipeline run is grouped under one build in the dashboard.

testng-grid.xml uses parallel="tests" with one <test> per browser. How do the tests inside the Chrome block run?

With tests mode the browsers run side by side, but each block is one thread.

Parallel limits and queueing#

Every grid has a ceiling on concurrent sessions. On TestingBot that limit comes with your plan, and according to the docs, sessions beyond it are queued for up to 6 minutes and start as soon as a slot is free. Set thread-count to roughly your limit. A much higher number does not make the run faster, and long queue waits can push slow tests into timeouts.

Testing a site that is not public#

A cloud browser cannot reach localhost:3000 or a staging server behind your firewall. TestingBot's tunnel solves this: start it on a machine that can reach the site,

java -jar testingbot-tunnel.jar key secret

then point RemoteWebDriver at the tunnel instead of the public hub, by default http://localhost:4445/wd/hub when the tunnel runs on the same machine as the tests. Your tests can then open internal URLs as if the browser were on your network. The tunnel documentation covers installation and options.

The grid browsers cannot reach your staging server behind the firewall. With TestingBot Tunnel running on the same machine as the tests, where should RemoteWebDriver point?

Commands go through the local tunnel endpoint, which routes the remote browser's traffic back through your network.

Debugging a remote failure#

Remote failures feel harder to debug because you cannot watch the browser. In practice you have more evidence than locally:

  • Your listener's screenshot still works: TakesScreenshot is a WebDriver command, so it captures the remote browser.
  • The session's video (recorded by default) and logs are attached to the test in the dashboard, under the name you set in tb:options.
  • The session ID, ((RemoteWebDriver) driver()).getSessionId(), ties a TestNG result to the exact remote session; log it in onTestFailure.

Start by watching the video around the failing command. Most "it only fails on the grid" problems turn out to be window size, a slower page load that an explicit wait would have covered, or a browser-specific behaviour you would never have seen with Chrome alone, which is the reason to run on several browsers in the first place.