Suites, groups and parameters in testng.xml

Annotations describe individual tests. testng.xml describes a run: which classes take part, which groups are included, which browser to use, how many threads, which listeners to attach. Keeping that in a file instead of in code means CI can run a smoke suite on every commit and a full regression suite nightly from the same codebase, just by pointing at a different XML file.

Anatomy of a suite file#

<!DOCTYPE suite SYSTEM "https://testng.org/testng-1.0.dtd">
<suite name="Regression" verbose="1">

  <test name="Web form - Chrome">
    <parameter name="browser" value="chrome"/>
    <classes>
      <class name="com.example.FirstTest"/>
      <class name="com.example.FormDefaultsTest"/>
      <class name="com.example.WebFormDataTest"/>
    </classes>
  </test>

</suite>

• <suite> is the whole run. It holds suite-wide settings such as parallel, thread-count and <listeners>.
• <test> is a named block of classes with its own parameters and group selection. This is the level @BeforeTest / @AfterTest refer to. A suite can contain many <test> blocks, for example one per browser.
• <classes> lists classes explicitly. Alternatively, <packages><package name="com.example.checkout"/></packages> picks up every test class in a package, so new classes join the suite automatically.

Within a class you can narrow down to specific methods:

<class name="com.example.WebFormDataTest">
  <methods>
    <include name="submitsAnyText"/>
  </methods>
</class>

Use method-level includes for short-term debugging only; long-term, groups scale better.

Groups#

Tag tests with one or more groups in the annotation:

@Test(groups = "smoke")
public void formPageLoads() { ... }

@Test(groups = {"regression", "forms"})
public void formDefaults() { ... }

Then select them in the suite:

<test name="Smoke">
  <groups>
    <run>
      <include name="smoke"/>
      <exclude name="quarantine"/>
    </run>
  </groups>
  <packages>
    <package name="com.example"/>
  </packages>
</test>

Exclusion wins over inclusion, which makes a quarantine group a practical way to park a known-flaky test without deleting it, while the ticket to fix it is open. Keep the list short and reviewed; a quarantine group that only grows is a test suite slowly switching itself off.

Remember the rule from chapter 3: once you filter by groups, configuration methods without a matching group are skipped unless they have alwaysRun = true. That is why BaseTest uses it.

You can also pick groups from the command line. With the Surefire 3.5.x setup from chapter 1, this runs only smoke tests from the default suite file:

mvn test -Dgroups=smoke

Parameters: configuration without code changes#

<parameter> values flow into methods annotated with @Parameters. The most useful one for a Selenium suite is the browser. Update BaseTest:

package com.example;

import org.openqa.selenium.WebDriver;
import org.openqa.selenium.chrome.ChromeDriver;
import org.openqa.selenium.chrome.ChromeOptions;
import org.openqa.selenium.firefox.FirefoxDriver;
import org.openqa.selenium.firefox.FirefoxOptions;
import org.testng.annotations.AfterMethod;
import org.testng.annotations.BeforeMethod;
import org.testng.annotations.Optional;
import org.testng.annotations.Parameters;

public abstract class BaseTest {

    private WebDriver driver;

    protected WebDriver driver() {
        return driver;
    }

    @BeforeMethod(alwaysRun = true)
    @Parameters("browser")
    public void startDriver(@Optional("chrome") String browser) {
        boolean headless = Boolean.getBoolean("headless");
        if ("firefox".equals(browser)) {
            FirefoxOptions options = new FirefoxOptions();
            if (headless) options.addArguments("-headless");
            driver = new FirefoxDriver(options);
        } else {
            ChromeOptions options = new ChromeOptions();
            if (headless) options.addArguments("--headless=new", "--window-size=1920,1080");
            driver = new ChromeDriver(options);
        }
    }

    @AfterMethod(alwaysRun = true)
    public void stopDriver() {
        if (driver != null) {
            driver.quit();
            driver = null;
        }
    }
}

(The waitUpTo helper from chapter 3 stays as it was; it is omitted here for brevity.)

@Optional("chrome") supplies a default when no <parameter name="browser"> is in scope, for instance when you run a single class from the IDE or with -Dtest. Without it, TestNG fails the configuration method because the parameter is missing, and every test in the class is skipped.

Parameters are inherited downward: a <parameter> on <suite> applies to every <test> unless a <test> defines the same name. That gives a clean cross-browser suite:

<suite name="Cross-browser">
  <parameter name="baseUrl" value="https://www.selenium.dev/selenium/web/"/>

  <test name="Chrome">
    <parameter name="browser" value="chrome"/>
    <packages><package name="com.example"/></packages>
  </test>

  <test name="Firefox">
    <parameter name="browser" value="firefox"/>
    <packages><package name="com.example"/></packages>
  </test>
</suite>

Parameters are strings and are fixed when the suite starts. For secrets such as passwords or API keys, read environment variables in code (System.getenv("TB_KEY")) instead; suite files end up in version control and in CI logs.

Parameters and data providers solve different problems. A parameter configures the environment of the whole run (which browser, which base URL). A data provider varies the inputs of one test. If you find yourself writing a data provider that returns browser names, a <test> block per browser is usually clearer.

Dependencies between tests#

Sometimes a later test is pointless if an earlier one failed. dependsOnMethods makes that explicit:

@Test
public void formPageLoads() { ... }

@Test(dependsOnMethods = "formPageLoads")
public void formDefaults() { ... }

If formPageLoads fails, formDefaults is reported as skipped, not failed, so your report shows one real failure instead of a cascade. dependsOnGroups does the same for whole groups (for example, everything depends on a health group that checks the site is up). Adding alwaysRun = true to the dependent test makes it run anyway but still in order.

Use dependencies to avoid noise, not to share state. If formDefaults relies on formPageLoads having navigated the browser somewhere, the two are really one test, and they will break as soon as each gets its own browser in parallel runs.

Several suite files#

A typical project ends up with a few suites:

testng.xml            full regression (default)
testng-smoke.xml      smoke group only, fast feedback on every commit
testng-grid.xml       cross-browser blocks for the remote grid (chapter 9)

Select one at run time through the suiteXmlFile property defined in the pom:

mvn test -DsuiteXmlFile=testng-smoke.xml

A suite file can also include others with <suite-files><suite-file path="testng-smoke.xml"/></suite-files>, which is handy for a "run everything" entry point.

If you would like your tests written as Gherkin scenarios instead, Cucumber has a TestNG runner that plugs into the same suite mechanics; the TestNG + Cucumber guide shows the setup. The TestNG guide in our docs uses exactly this parameter-per-<test> pattern for cross-browser runs, which we will build on in chapter 9.

Next, the suite file gets the two attributes everyone wants first: parallel and thread-count.