Listeners, screenshots and reports

A failing UI test with only a stack trace leaves you guessing what the page looked like. TestNG listeners let you run code on every test event without touching the tests themselves, which makes them the right place for screenshots on failure, extra logging and custom reporting.

ITestListener in one minute#

org.testng.ITestListener has a callback for each stage of a test's life. All methods have default empty implementations, so you override only the ones you need:

ITestResult carries what you need to know about the test: getName(), getMethod(), getTestClass(), getParameters() (the data provider row), getThrowable() (the failure), getStartMillis() and getEndMillis().

Screenshot on failure#

With the ThreadLocal base class from chapter 7, a listener can reach the current thread's driver through BaseTest.driver():

package com.example;

import java.io.IOException;
import java.nio.file.Files;
import java.nio.file.Path;
import org.openqa.selenium.OutputType;
import org.openqa.selenium.TakesScreenshot;
import org.openqa.selenium.WebDriver;
import org.testng.ITestListener;
import org.testng.ITestResult;
import org.testng.Reporter;

public class ScreenshotListener implements ITestListener {

    @Override
    public void onTestFailure(ITestResult result) {
        WebDriver driver = BaseTest.driver();
        if (driver == null) {
            return;   // failed before a browser existed
        }
        byte[] png = ((TakesScreenshot) driver).getScreenshotAs(OutputType.BYTES);
        Path file = Path.of("target", "screenshots",
                result.getTestClass().getRealClass().getSimpleName() + "-"
                        + result.getMethod().getMethodName() + "-"
                        + result.getStartMillis() + ".png");
        try {
            Files.createDirectories(file.getParent());
            Files.write(file, png);
            Reporter.log("Screenshot: " + file.toAbsolutePath(), true);
        } catch (IOException e) {
            Reporter.log("Could not save screenshot: " + e.getMessage(), true);
        }
    }
}

Points worth noticing:

• Timing. This only works if the browser is still open when the listener runs. We checked the order with TestNG 7.12.0: for a failing test, onTestFailure runs before the @AfterMethod that quits the driver, so the screenshot shows the page as it was at the moment of failure.
• Same thread. The listener callback runs on the test's thread, so BaseTest.driver() returns the right browser even under parallel="methods".
• Unique file names. Data-driven tests have the same method name for every row, and in parallel runs several can fail at once. Including the start time keeps one row's screenshot from overwriting another's. Our first version used just class and method name and lost three of four screenshots exactly that way.
• OutputType.BYTES avoids a temporary file and an extra dependency for copying it. OutputType.FILE plus a copy works too.
• Reporter.log(message, true) writes the message into TestNG's HTML reports for that test and, with true, to the console as well.

A listener instance is shared by all threads, so don't keep per-test state in its fields. If you need to, use a ConcurrentHashMap keyed by ITestResult or a ThreadLocal.

Registering a listener#

There are three ways, from most local to most global:

1. On a class, for that class (and in practice, the whole suite: TestNG registers listeners declared with @Listeners globally):

import org.testng.annotations.Listeners;

@Listeners(ScreenshotListener.class)
public class FirstTest extends BaseTest { ... }

2. In testng.xml, which keeps tests free of reporting concerns and is what we recommend:

<suite name="Course suite" parallel="methods" thread-count="3">
  <listeners>
    <listener class-name="com.example.ScreenshotListener"/>
  </listeners>
  ...
</suite>

3. Via Java's ServiceLoader: list the class name in src/test/resources/META-INF/services/org.testng.ITestNGListener. TestNG then picks it up for every run, including mvn test -Dtest=..., which ignores the suite file. Useful for a company-wide listener packaged in a shared library.

The reports TestNG already writes#

Run mvn test and look in target/surefire-reports:

index.html               TestNG's interactive report: suites, groups, times, output
emailable-report.html    a single self-contained summary page
testng-results.xml       TestNG's machine-readable results
junitreports/            JUnit-style XML per class
TEST-TestSuite.xml       Surefire's JUnit-style XML for the whole run

CI servers (GitHub Actions test reporters, Jenkins, GitLab) understand the JUnit-style XML, so point them at target/surefire-reports/*.xml and archive target/screenshots as a build artifact. Third-party reporters such as Allure and ExtentReports plug in the same way as our screenshot listener: as a TestNG listener added to the suite.

Retrying flaky tests, carefully#

TestNG can re-run a failed test through an IRetryAnalyzer:

package com.example;

import org.testng.IRetryAnalyzer;
import org.testng.ITestResult;

public class RetryOnce implements IRetryAnalyzer {
    private int attempts = 0;

    @Override
    public boolean retry(ITestResult result) {
        return attempts++ < 1;   // retry a failed test one time
    }
}

@Test(retryAnalyzer = RetryOnce.class)
public void submitsTheWebForm() { ... }

Listeners see the failed first attempt as a skip (onTestSkipped), TestNG's summary counts it under "Retries", and the final attempt is reported as passed or failed. When we ran two failing methods and a two-row data provider with this analyzer on TestNG 7.12.0, every method and every row was retried exactly once, so the counter can safely live in a field.

Use retries as a short-term bandage with a ticket attached, not as a strategy. A test that passes on the second attempt is telling you about a race condition, an environment problem or a real intermittent bug, and retries hide all three. Watch your reports for tests that are frequently retried, and fix the wait or the test data instead. Our article on exception handling with Selenium covers the exceptions that most often sit behind flaky failures.

Logging what the test did#

Two cheap additions make failures much easier to read:

@Override
public void onTestStart(ITestResult result) {
    Reporter.log("START " + result.getMethod().getQualifiedName()
            + " params=" + java.util.Arrays.toString(result.getParameters()), true);
}

and, in the failure handler, the current URL and page title (driver.getCurrentUrl(), driver.getTitle()), which often reveal an unexpected redirect to a login or error page before you even open the screenshot.

Local screenshots cover the moment of failure. When tests run on a remote grid, you usually also get a video and logs of the whole session; the next chapter moves the suite there, and the TestingBot test options list how to control recording.