Exit course
Chapter 8 of 10 Intermediate 9 min read 3 quick checks Updated

Run the Framework in Parallel on a Grid

On this page
  1. What breaks: a shared driver field
  2. The fix: one driver per thread
  3. Python and Playwright: processes, not threads
  4. Pointing the framework at a remote grid
  5. Sizing the parallelism

A UI suite that takes 40 minutes runs less often, and a suite that runs less often catches bugs later. Parallelism is the main lever, and it is where framework design gets tested: every shortcut you took with shared state shows up as a random failure the first time two tests run at once. This chapter makes the Java framework thread-safe, explains why Python and Playwright mostly avoid the problem, and then points all three at a remote grid.

What breaks: a shared driver field#

Chapter 01's Java test kept the driver in an instance field and created it in @BeforeMethod. Here is what happens when you run four such tests with parallel="methods" and four threads. We ran this against the practice site; this is the actual output of one run:

SharedFieldTest.a:15 expected [Your password is invalid!] but found [Your username is invalid!]
SharedFieldTest.c:17 >> NoSuchSession Session ID is null. Using WebDriver after calling quit()?
SharedFieldTest.d:18 expected [Your password is invalid!] but found [Your username is invalid!]
SharedFieldTest.stop:13 >> WebDriver Timed out waiting for driver server to stop.
Tests run: 5, Failures: 4

TestNG creates one instance of the test class and calls its methods from several threads. Each @BeforeMethod overwrites the same driver field, so test a typed into the browser that test b was also using and read b's error message. Test c used a driver that another thread had already quit. The second run of the same suite produced a different, smaller set of failures, which is what makes this kind of bug so expensive: it looks like flakiness, not like a design error. Browsers that lost their field reference were also never quit.

The fix: one driver per thread#

A ThreadLocal gives each thread its own slot. The DriverFactory from chapter 06 is built on one:

package com.example.framework;

import java.net.MalformedURLException;
import java.net.URI;
import java.util.HashMap;
import java.util.Map;

import org.openqa.selenium.MutableCapabilities;
import org.openqa.selenium.WebDriver;
import org.openqa.selenium.chrome.ChromeDriver;
import org.openqa.selenium.chrome.ChromeOptions;
import org.openqa.selenium.firefox.FirefoxDriver;
import org.openqa.selenium.firefox.FirefoxOptions;
import org.openqa.selenium.remote.RemoteWebDriver;

public final class DriverFactory {
    private static final ThreadLocal<WebDriver> DRIVER = new ThreadLocal<>();

    private DriverFactory() {}

    public static WebDriver start(String testName) {
        WebDriver driver = "testingbot".equals(Config.target())
                ? remote(testName)
                : local();
        DRIVER.set(driver);
        return driver;
    }

    public static WebDriver get() {
        WebDriver driver = DRIVER.get();
        if (driver == null) {
            throw new IllegalStateException("No driver for this thread. Did @BeforeMethod run?");
        }
        return driver;
    }

    public static void quit() {
        WebDriver driver = DRIVER.get();
        if (driver != null) {
            try {
                driver.quit();
            } finally {
                DRIVER.remove();   // never leave a dead driver on a pooled thread
            }
        }
    }

    private static WebDriver local() {
        if ("firefox".equals(Config.browser())) {
            return new FirefoxDriver(new FirefoxOptions());
        }
        ChromeOptions options = new ChromeOptions();
        options.addArguments("--window-size=1280,900");
        return new ChromeDriver(options);
    }

    private static WebDriver remote(String testName) {
        MutableCapabilities options = "firefox".equals(Config.browser())
                ? new FirefoxOptions()
                : new ChromeOptions();
        options.setCapability("platformName", "WIN11");
        options.setCapability("browserVersion", "latest");

        Map<String, Object> tbOptions = new HashMap<>();
        tbOptions.put("name", testName);
        tbOptions.put("build", System.getenv().getOrDefault("BUILD_ID", "local-build"));
        options.setCapability("tb:options", tbOptions);

        String hub = "https://" + System.getenv("TB_KEY") + ":" + System.getenv("TB_SECRET")
                + "@hub.testingbot.com/wd/hub";
        try {
            return new RemoteWebDriver(URI.create(hub).toURL(), options);
        } catch (MalformedURLException e) {
            throw new IllegalStateException("Bad hub URL", e);
        }
    }
}

Rules that keep it safe:

  • Tests never store the driver in a field. They call driver() from BaseTest, which reads the current thread's slot. Page objects receive the driver in their constructor and are created inside the test method, so each page object belongs to one thread.
  • remove() after quit(). TestNG reuses pool threads. Without remove(), the next test on that thread could see a quit driver instead of the clear "No driver for this thread" error.
  • No static page objects or static WebElements. A static LoginPage is a shared field with extra steps.
  • @BeforeMethod, not @BeforeClass, for the driver, when you run parallel="methods". With parallel="classes" a per-class driver can work, but then methods within a class run sequentially.

Then switch on parallelism in testng.xml:

<!DOCTYPE suite SYSTEM "https://testng.org/testng-1.0.dtd">
<suite name="login-suite" parallel="methods" thread-count="4">
  <test name="login">
    <classes>
      <class name="com.example.tests.LoginTest"/>
      <class name="com.example.tests.LoginErrorsTest"/>
      <class name="com.example.tests.NavigationTest"/>
    </classes>
  </test>
</suite>

One Maven gotcha we hit while testing this: maven-surefire-plugin 3.6.0 and later run TestNG on the JUnit Platform and reject the suiteXmlFiles setting with an error. If your build runs suites from testng.xml, pin surefire to the 3.5.x line (3.5.6 at the time of writing) until you migrate to groups or includes.

With the ThreadLocal factory, all eight tests in our suite pass at thread-count="4", run after run.

Why does DriverFactory.quit() call DRIVER.remove() after driver.quit()?

TestNG reuses threads. Clearing the slot means a missing @BeforeMethod fails with 'No driver for this thread' rather than a confusing session error.

Python and Playwright: processes, not threads#

pytest-xdist runs tests in separate worker processes:

pip install pytest-xdist==3.8.0
pytest -n 4          # or -n auto for one worker per CPU core

Each process has its own interpreter, so a function-scoped driver fixture cannot be shared across workers by accident and no ThreadLocal is needed. The things that can still collide are outside the process: the same user account edited by two tests, files written to the same path, or a scope="session" fixture that you expected to run once (it runs once per worker).

Playwright Test also uses worker processes. fullyParallel: true lets tests within one file run in parallel, and workers sets how many processes run at once. Every test gets its own browser context from the page fixture, and our page objects are created per test by the loginPage fixture, so they are isolated by construction. If you add a worker-scoped fixture (for example, a logged-in storage state), remember it is shared by every test in that worker.

You add a worker-scoped Playwright fixture that holds a logged-in storage state. Which tests share it?

Worker scope means one value per worker process, reused by every test that worker runs. That is fine for read-only state and a collision risk for anything a test changes.

Pointing the framework at a remote grid#

A local machine runs a handful of browsers before CPU becomes the bottleneck, and it only has the browsers installed on it. A remote Selenium Grid or cloud grid removes both limits, and with the factory in place, switching is config, not code.

Java: set target=testingbot in the environment file (or -Dtarget=testingbot) and export TB_KEY / TB_SECRET. The factory above creates a RemoteWebDriver against https://hub.testingbot.com/wd/hub, as described in the TestingBot Java documentation, with the TestNG method name as the session name and a build so the parallel sessions from one run group together.

To show pass/fail on the grid, report the TestNG result in BaseTest with the TestingBot Java REST client (com.testingbot:testingbotrest:1.1.0):

@AfterMethod(alwaysRun = true)
public void stopDriver(ITestResult result) {
    try {
        if ("testingbot".equals(Config.target())) {
            String sessionId = ((RemoteWebDriver) DriverFactory.get()).getSessionId().toString();
            TestingbotREST api = new TestingbotREST(System.getenv("TB_KEY"), System.getenv("TB_SECRET"));
            Map<String, Object> data = new HashMap<>();
            data.put("success", result.isSuccess() ? "1" : "0");
            api.updateTest(sessionId, data);
        }
    } finally {
        DriverFactory.quit();
    }
}

Python: the remote branch of the driver fixture:

options = webdriver.ChromeOptions()
options.set_capability("platformName", "WIN11")
options.set_capability("browserVersion", "latest")
options.set_capability("tb:options", {
    "key": os.environ["TB_KEY"],
    "secret": os.environ["TB_SECRET"],
    "name": request.node.name,
})
drv = webdriver.Remote(
    command_executor="https://hub.testingbot.com/wd/hub",
    options=options,
)

(Add request to the fixture's parameters to get the test name.) Then TARGET=testingbot pytest -n 4.

Playwright does not use the Selenium hub. It connects to a remote browser over a WebSocket, set per project with connectOptions:

// testingbot.ts
export function testingBotEndpoint(caps: Record<string, unknown>): string {
  const capabilities = {
    'tb:options': {
      key: process.env.TB_KEY,
      secret: process.env.TB_SECRET,
      build: process.env.BUILD_ID ?? 'local-build',
    },
    ...caps,
  };
  return `wss://cloud.testingbot.com/playwright?capabilities=${encodeURIComponent(
    JSON.stringify(capabilities),
  )}`;
}

// playwright.config.ts (projects when TARGET=testingbot)
projects: [
  {
    name: 'chrome-win11',
    use: {
      connectOptions: {
        wsEndpoint: testingBotEndpoint({ browserName: 'chrome', browserVersion: 'latest', platform: 'WIN11' }),
      },
    },
  },
  {
    name: 'firefox-linux',
    use: {
      connectOptions: {
        wsEndpoint: testingBotEndpoint({ browserName: 'firefox', browserVersion: 'latest', platform: 'LINUX' }),
      },
    },
  },
],

Because the page objects only depend on page, nothing else changes: the same specs now run on two browser/OS combinations. Playwright's protocol does not carry a test result, so add an automatic fixture that reports it with the testingbot_executor command from the mark test status documentation:

reportStatus: [
  async ({ page }, use, testInfo) => {
    await use();
    if (process.env.TARGET !== 'testingbot') return;
    const passed = testInfo.status === testInfo.expectedStatus;
    const payload = JSON.stringify({
      action: 'setSessionStatus',
      arguments: { passed, reason: testInfo.error?.message?.slice(0, 200) ?? 'ok' },
    });
    await page.evaluate(() => {}, `testingbot_executor: ${payload}`);
  },
  { auto: true },
],

Add reportStatus: void to the fixture types; { auto: true } runs it for every test without any test asking for it.

Sizing the parallelism#

Set thread-count, -n or workers to the number of parallel sessions your grid allows, not to what your laptop can handle: the browsers no longer run locally. On TestingBot, sessions beyond your plan's parallel limit are queued (for up to 6 minutes, per the Java docs) and start when a slot frees up, so oversubscribing slightly does not fail the run, but it does not make it faster either. More on how parallel sessions are counted is on the parallel testing page.

Your grid plan allows five parallel sessions and your laptop has 12 cores. What thread-count should a remote run use?

The browsers no longer run locally, so the plan's parallel limit is the ceiling. Extra sessions are queued and do not make the run faster.