Data-driven tests with DataProvider

Many UI tests are the same flow with different inputs: a search with ten queries, a sign-up form with valid and invalid emails, a checkout in three currencies. Copy-pasting the test ten times creates ten things to maintain. A TestNG @DataProvider keeps one test method and feeds it rows of data; each row becomes its own test invocation with its own result.

A first data provider#

package com.example;

import org.openqa.selenium.By;
import org.openqa.selenium.support.ui.ExpectedConditions;
import org.testng.Assert;
import org.testng.annotations.DataProvider;
import org.testng.annotations.Test;

public class WebFormDataTest extends BaseTest {

    private static final String FORM = "https://www.selenium.dev/selenium/web/web-form.html";

    @DataProvider(name = "textInputs")
    public Object[][] textInputs() {
        return new Object[][] {
            {"plain ascii"},
            {"with spaces and 123"},
            {"Ünïcödé"},
            {""},
        };
    }

    @Test(dataProvider = "textInputs")
    public void submitsAnyText(String text) {
        driver().get(FORM);
        driver().findElement(By.id("my-text-id")).sendKeys(text);
        driver().findElement(By.cssSelector("button[type='submit']")).click();

        String message = waitUpTo(10)
                .until(ExpectedConditions.visibilityOfElementLocated(By.id("message")))
                .getText();
        Assert.assertEquals(message, "Received!");
    }
}

Each inner array is one invocation; its elements are matched to the test method's parameters in order. Four rows, four test results, and each gets a fresh browser because @BeforeMethod runs before every invocation. The report shows the parameters with each result, for example submitsAnyText[plain ascii], so a failure tells you which row broke.

Three behaviours to know (we checked each against TestNG 7.12.0):

• The values in a row must match the method's parameters. If they don't, the test fails with a MethodMatcherException instead of running.
• If the data provider method itself throws (a missing file, say), the test is reported as failed with the provider's exception as the cause.
• An empty provider means zero invocations. Nothing fails; the test simply does not appear in the results, which is easy to miss when a data file is accidentally emptied.

Multiple parameters and expected results#

Put the expected outcome in the row, next to the input:

@DataProvider(name = "emails")
public Object[][] emails() {
    return new Object[][] {
        // input                 valid?
        {"qa@example.com",       true},
        {"qa+tag@example.co.uk", true},
        {"no-at-sign.example",   false},
        {"",                     false},
    };
}

@Test(dataProvider = "emails")
public void validatesEmail(String email, boolean shouldBeValid) {
    // fill the form, submit, then assert on shouldBeValid
}

Mixing positive and negative cases in one provider is fine as long as the expected result is part of the data. If the two branches of the test diverge much, split them into two providers and two methods; a test full of if (shouldBeValid) is hard to read.

Sharing providers between classes#

Put shared data in its own class and reference it with dataProviderClass. The provider method must be static when it lives in another class:

package com.example;

import org.testng.annotations.DataProvider;

public final class TestData {
    private TestData() {}

    @DataProvider(name = "browsersizes")
    public static Object[][] browserSizes() {
        return new Object[][] {
            {375, 812},
            {768, 1024},
            {1920, 1080},
        };
    }
}

@Test(dataProvider = "browsersizes", dataProviderClass = TestData.class)
public void headerFitsAt(int width, int height) {
    driver().manage().window().setSize(new org.openqa.selenium.Dimension(width, height));
    // ...
}

Reading rows from a file#

Hard-coded arrays are fine for a handful of rows. For more, keep the data in src/test/resources and return an Iterator<Object[]>, which TestNG consumes lazily:

# src/test/resources/search-terms.csv
selenium,true
testng,true
zzzzqqq,false

import java.io.IOException;
import java.io.InputStream;
import java.nio.charset.StandardCharsets;
import java.util.Iterator;
import java.util.List;
import org.testng.annotations.DataProvider;

@DataProvider(name = "searchTerms")
public static Iterator<Object[]> searchTerms() throws IOException {
    try (InputStream in = TestData.class.getResourceAsStream("/search-terms.csv")) {
        List<String> lines = new String(in.readAllBytes(), StandardCharsets.UTF_8).lines().toList();
        return lines.stream()
                .filter(line -> !line.isBlank() && !line.startsWith("#"))
                .map(line -> line.split(",", -1))
                .map(cols -> new Object[] {cols[0], Boolean.parseBoolean(cols[1])})
                .iterator();
    }
}

The file is read fully inside the try block, so closing the stream does not break the iterator. For real CSV with quoting and escapes, use a CSV library rather than split.

Keep test data next to the tests, in version control, and small. Data that changes per environment (base URLs, account names) belongs in configuration, covered by testng.xml parameters in the next chapter.

Knowing which test is asking#

A provider can declare a java.lang.reflect.Method parameter, and TestNG passes in the test method that is requesting data. That lets one provider serve several tests:

@DataProvider(name = "accounts")
public Object[][] accounts(java.lang.reflect.Method method) {
    if (method.getName().contains("Admin")) {
        return new Object[][] {{"admin@example.com"}};
    }
    return new Object[][] {{"user1@example.com"}, {"user2@example.com"}};
}

Use it sparingly: two plainly named providers are usually easier to follow.

Running only some rows#

While debugging one failing row, the indices attribute restricts a provider to specific positions (zero-based) without editing the data:

@DataProvider(name = "textInputs", indices = {2})   // only the third row
public Object[][] textInputs() { ... }

Remember to remove it again; a committed indices silently shrinks your coverage.

Parallel data providers#

By default the rows run one after another. With parallel = true, TestNG runs the invocations of that test concurrently:

@DataProvider(name = "textInputs", parallel = true)
public Object[][] textInputs() { ... }

Parallel data providers use their own thread pool, sized by data-provider-thread-count in testng.xml (default 10), separate from the suite's thread-count (default 5):

<suite name="Course suite" data-provider-thread-count="4">

When we ran the four-row provider above with data-provider-thread-count="2", the console showed the rows split across two pool threads:

TestNG-PoolService-1 -> [plain ascii]
TestNG-PoolService-2 -> [with spaces and 123]
TestNG-PoolService-2 -> []
TestNG-PoolService-1 -> [Ünïcödé]

Do not switch this on with the BaseTest from chapter 3. All invocations of a data-driven test share one instance of the test class, so they share its single driver field. Two threads would overwrite each other's browser and quit each other's sessions, producing failures that look random. Chapter 7 moves the driver into a ThreadLocal, after which parallel = true is safe.

Also size the pool for where the browsers run. Ten parallel Chromes on a laptop will slow each other down until waits time out; on a remote grid, the limit is the number of parallel sessions your plan allows, as covered on our parallel testing page.

For a broader introduction to how TestNG data providers compare with JUnit's parameterised tests, see the TestNG with Selenium article.