Test Automation Framework Structure

A test automation framework is not a library you download. It is the set of conventions and small classes that decide how every test in your repository gets a browser, finds its data, reports its result and cleans up. With page objects, config and test data in place, this chapter fixes the layout, the base classes, and reporting, then looks hard at retries.

Layers, and which way they depend#

tests            what to check            (assertions live here)
  |
pages/components what the UI offers       (locators and user actions)
  |
framework        how to drive a browser   (driver factory, config, waits, reporting)
  |
data             who and with what        (users, builders, data providers)

Dependencies point down only. Tests import pages and data; pages import the framework's helpers; the framework imports nothing from tests or pages. When a page object imports a test class, or the driver factory knows about LoginPage, the layering has broken and changes will start rippling across the suite.

Folder layout#

Java (Maven, TestNG):

src/test/java/com/example/
  components/   SiteHeader.java
  data/         User.java
  framework/    Config.java, DriverFactory.java, ScreenshotOnFailure.java
  pages/        BasePage.java, LoginPage.java, LoggedInPage.java, PracticeIndexPage.java
  tests/        BaseTest.java, LoginTest.java, LoginErrorsTest.java, NavigationTest.java
src/test/resources/env/
  practice.properties, staging.properties
testng.xml
pom.xml

Python (pytest):

config/       practice.toml, staging.toml
pages/        __init__.py, base_page.py, login_page.py
tests/        test_login.py, test_login_errors.py
conftest.py   driver, settings and reporting fixtures
settings.py

Playwright (TypeScript):

components/   site-header.ts
config/       env.ts
pages/        login-page.ts, logged-in-page.ts, practice-index-page.ts
tests/        fixtures.ts, login.spec.ts, login-errors.spec.ts, navigation.spec.ts
playwright.config.ts
testingbot.ts

Group by layer first, then by feature once a layer has more than a dozen files (pages/checkout/, pages/account/). Mirroring the application's navigation in the folder tree makes the right page object easy to find for someone new.

The driver factory and base test#

Tests should not create browsers. In Java, a DriverFactory owns that decision (local Chrome, local Firefox, or a remote grid, read from Config) and a BaseTest calls it around every test method:

package com.example.tests;

import java.lang.reflect.Method;

import org.openqa.selenium.WebDriver;
import org.testng.annotations.AfterMethod;
import org.testng.annotations.BeforeMethod;

import com.example.framework.DriverFactory;

public abstract class BaseTest {
    @BeforeMethod
    public void startDriver(Method method) {
        DriverFactory.start(getClass().getSimpleName() + "." + method.getName());
    }

    @AfterMethod(alwaysRun = true)
    public void stopDriver() {
        DriverFactory.quit();
    }

    protected WebDriver driver() {
        return DriverFactory.get();
    }
}

TestNG injects the Method into @BeforeMethod, so the factory receives a readable test name it can pass to a remote grid as the session name. alwaysRun = true makes sure the browser is quit even when setup of a later step failed. The factory itself, and why driver() is a method call rather than a field, is the subject of chapter 07.

In pytest, the driver fixture in conftest.py plays both roles, and in Playwright the built-in page fixture does. Neither language needs a base test class, which is a hint: inheritance is only one way to share setup, and in Java it is the conventional one.

Keep BaseTest this small. A base test with 40 helper methods is the test-side version of the god object discussed in chapter 08.

Reporting: make failures self-explaining#

A red test in CI is only useful if someone can tell why from the report, without re-running it. Every framework needs, at minimum, a machine-readable result file for CI and a screenshot at the moment of failure.

TestNG writes JUnit-style XML into target/surefire-reports/. Screenshots come from a listener:

package com.example.framework;

import java.io.IOException;
import java.nio.file.Files;
import java.nio.file.Path;

import org.openqa.selenium.OutputType;
import org.openqa.selenium.TakesScreenshot;
import org.testng.ITestListener;
import org.testng.ITestResult;

public class ScreenshotOnFailure implements ITestListener {
    @Override
    public void onTestFailure(ITestResult result) {
        try {
            byte[] png = ((TakesScreenshot) DriverFactory.get()).getScreenshotAs(OutputType.BYTES);
            Path dir = Path.of("target", "screenshots");
            Files.createDirectories(dir);
            Files.write(dir.resolve(result.getName() + ".png"), png);
        } catch (IOException | RuntimeException e) {
            System.err.println("Could not save screenshot: " + e.getMessage());
        }
    }
}

Register it with @Listeners(ScreenshotOnFailure.class) on BaseTest, or in testng.xml. onTestFailure runs before @AfterMethod, so the browser is still open. The catch matters: a reporting failure must never hide the real test failure.

pytest writes JUnit XML with pytest --junitxml=reports/junit.xml. For screenshots, wrap the report hook in conftest.py:

from pathlib import Path

import pytest


@pytest.hookimpl(wrapper=True)
def pytest_runtest_makereport(item, call):
    report = yield
    if report.when == "call" and report.failed:
        drv = item.funcargs.get("driver")
        if drv is not None:
            out = Path("reports/screenshots")
            out.mkdir(parents=True, exist_ok=True)
            drv.save_screenshot(str(out / f"{item.name}.png"))
    return report

Playwright has the richest built-in story. With reporter: [['list'], ['html', { open: 'never' }]] and trace: 'retain-on-failure' in use, every failed test keeps a trace you can open with npx playwright show-trace: DOM snapshots before and after each action, network, console and the source line. Add ['junit', { outputFile: 'results.xml' }] to the reporter list for CI.

Retries, and the case against them#

Every runner can retry failed tests:

// playwright.config.ts
retries: process.env.CI ? 1 : 0,

package com.example.framework;

import org.testng.IRetryAnalyzer;
import org.testng.ITestResult;

public class RetryOnce implements IRetryAnalyzer {
    private int attempts = 0;

    @Override
    public boolean retry(ITestResult result) {
        if (attempts < 1) {
            attempts++;
            return true;
        }
        return false;
    }
}

pip install pytest-rerunfailures
pytest --reruns 1

In TestNG, attach it with @Test(retryAnalyzer = RetryOnce.class). When we ran a deliberately failing test with it, TestNG reported the first attempt as skipped and the second as failed, so a retried-then-passed test looks like a pass with one skip. Playwright is more honest: a test that fails and then passes is reported as flaky, separately from passed.

Retries make the build green. They do not make the product or the test correct. The arguments against leaning on them:

• They hide real bugs. A race condition in the application (double-submit, a slow API that sometimes times out) fails one run in ten. A retry turns that signal into a green tick.
• They hide test bugs too. Most flaky UI tests have a concrete cause: a missing wait, shared data, test order dependency. The Selenium docs list "test dependency" among their discouraged practices for exactly this reason. Each of those has a fix; a retry postpones it.
• They cost time. A suite with a 5% flake rate and one retry runs noticeably longer, on every build, forever.
• They erode trust. Once people know "it usually passes on retry", they stop reading failures.

A reasonable policy:

1. Retries off locally (retries: 0). You want to see flakiness while you work.
2. At most one retry in CI, and only so a single flake does not block an unrelated merge.
3. Treat "flaky" as a failure that is tracked. In Playwright, failOnFlakyTests: !!process.env.CI in the config (or --fail-on-flaky-tests on the command line) fails the run if any test needed a retry; use it on your main branch even if pull requests allow retries.
4. Keep the trace or video from the first failing attempt. That is the evidence. trace: 'retain-on-failure' keeps it; on-first-retry only starts recording on the retry, which is cheaper but loses the original failure.
5. Fix or quarantine flaky tests within a fixed time. A quarantined test that nobody owns is a deleted test.

Our article on handling test flakiness goes deeper into root causes. For how TestingBot records the session video and logs for every remote test, see test results in the Selenium docs.