Assertions, screenshots on failure and test reports

A failing UI test is only useful if it tells you what went wrong. This chapter covers how pytest assertions report failures, how to save a screenshot of the browser at the moment a test fails, and how to produce HTML and JUnit XML reports for people and CI servers.

Plain assert#

pytest rewrites assert statements in test files so that a failure shows the values involved. Change the expected title in a test to "Web form!" and run it:

    def test_title_typo(driver):
        WebFormPage(driver).open()
>       assert driver.title == "Web form!"
E       AssertionError: assert 'Web form' == 'Web form!'
E
E         - Web form!
E         ?         -
E         + Web form

No assertEqual, no message string: pytest shows both sides and a diff. A few habits keep these messages useful:

• Assert on values, not booleans. assert driver.title == "Web form" reports both titles; assert driver.title == "Web form" is True and assert is_title_ok() only report False.
• Compare whole structures when you can. assert [b.get_attribute("id") for b in checked] == ["my-check-1", "my-check-2"] shows exactly which item differs.
• Use in for partial text. assert "my-select=2" in driver.current_url reads better than .find(...) != -1 and reports both strings.
• Add a message only when the values do not explain themselves: assert rows, "search returned no results for 'shoes'".
• Assert after waiting, not instead of waiting. If the text arrives asynchronously, wait with EC.text_to_be_present_in_element (chapter 3) and then assert. An assertion checks once.

pytest stops a test at its first failing assertion. When you want to check several independent things on one page and see every failure at once, split them into separate tests, or collect the problems and assert once at the end:

problems = []
if driver.title != "Web form":
    problems.append(f"title was {driver.title!r}")
if not driver.find_element(By.NAME, "my-readonly").get_attribute("readonly"):
    problems.append("read-only field is editable")
assert not problems, problems

Screenshots on failure#

The fixture's teardown runs after the test, while the browser is still open. If it knew whether the test failed, it could take a screenshot before calling quit(). pytest does not pass that information to fixtures directly, but a hook can attach it to the test item. Add this to conftest.py:

@pytest.hookimpl(wrapper=True, tryfirst=True)
def pytest_runtest_makereport(item, call):
    report = yield
    # Remember the outcome of each phase so fixtures can read it during teardown.
    setattr(item, "rep_" + report.when, report)
    return report

pytest builds a report for each of the three phases of a test, setup, call and teardown. The hook wraps that and stores each report on the item as rep_setup, rep_call and rep_teardown. By the time the fixture's teardown runs, request.node.rep_call holds the result of the test body.

Now extend the fixture's teardown:

import os
import re
from pathlib import Path


@pytest.fixture
def driver(request, browser):
    driver = ...  # created as before

    yield driver

    call = getattr(request.node, "rep_call", None)
    failed = call is None or call.failed
    try:
        if failed:
            save_screenshot(driver, request.node.nodeid)
    finally:
        driver.quit()


def save_screenshot(driver, nodeid):
    folder = Path("artifacts")
    folder.mkdir(exist_ok=True)
    worker = os.environ.get("PYTEST_XDIST_WORKER", "main")
    name = re.sub(r"[^\w.-]+", "_", nodeid)
    path = folder / f"{name}-{worker}.png"
    driver.save_screenshot(str(path))
    print(f"screenshot: {path}")

Details that matter:

• call is None covers the case where the test body never ran because another fixture failed during setup.
• try / finally around the screenshot means a failing screenshot (a browser that already crashed, for example) never prevents quit().
• The file name is built from the node ID, which includes the parameters ([chrome]), plus the pytest-xdist worker name. Without both, parallel runs and parametrised tests overwrite each other's screenshots.
• print output from teardown appears in the failure report, so the path is right next to the error.

Run the deliberately broken test again and the report ends with:

--------------------------- Captured stdout teardown ---------------------------
screenshot: artifacts/tests_test_broken.py_test_title_typo_chrome_-main.png

Add artifacts/ to .gitignore. In CI, upload that folder as a build artifact so the screenshots outlive the job.

HTML reports with pytest-html#

pytest-html (installed in chapter 1) turns a run into a single HTML page with every test, its duration, and the captured output and traceback of each failure:

pytest --headless --html=report.html --self-contained-html

--self-contained-html inlines the CSS so the file can be mailed or archived on its own. The screenshot paths printed by the fixture appear in each failed test's captured output.

JUnit XML for CI servers#

Jenkins, GitLab, Azure DevOps and most other CI servers read JUnit XML to show test results in their own UI. pytest writes it without any plugin:

pytest --headless --junitxml=junit.xml

You can combine both flags in one run. A common setup is JUnit XML for the CI server's test tab, the HTML report and artifacts/ folder as downloadable artifacts, and -ra in pytest.ini so the console log already lists what failed.

The tests are now well reported, but each one still spells out locators inline. When the form changes, every test that touches it has to change too. Chapter 6 gathers that knowledge into a page object.