Exit course
Chapter 9 of 11 Intermediate 7 min read 2 quick checks Updated

Debugging a Browser-Specific Failure

On this page
  1. The failing test
  2. Step 1: read what the failure actually says
  3. Step 2: watch the session
  4. Step 3: measure inside the browser
  5. Step 4: ask whether the test needed it
  6. When you need to see it yourself
  7. A checklist for one-browser failures
  8. Summary

Sooner or later the matrix shows the middle row from chapter 7: a test that passes everywhere except in one browser. This chapter walks through a real one from start to finish: a Selenium test that passed in Chrome on Windows and failed in Safari on macOS, how we narrowed it down, the fix that looked obvious and did not work, and the one that did.

The failing test#

The test types into a textarea on the Selenium project's web form, selects everything and types over it:

from selenium.webdriver.common.by import By
from selenium.webdriver.common.keys import Keys

FORM = "https://www.selenium.dev/selenium/web/web-form.html"


def test_replace_textarea_text(driver):
    driver.get(FORM)
    box = driver.find_element(By.NAME, "my-textarea")
    box.send_keys("draft")

    box.send_keys(Keys.CONTROL, "a")   # select all ... on Windows and Linux
    box.send_keys("final")

    assert box.get_property("value") == "final"

Run on two configurations from the matrix:

$ pytest tests/test_textarea.py -k "chrome-latest-WIN11 or safari"
>       assert box.get_property("value") == "final"
E       AssertionError: assert 'draft' == 'final'
FAILED tests/test_textarea.py::test_replace_textarea_text[safari-latest-TAHOE]
================== 1 failed, 1 passed, 3 deselected in 20.59s ==================

Step 1: read what the failure actually says#

Before opening anything, read the assertion. The value is draft, not finaldraft or draftfinal. So in Safari, the word final never reached the textarea at all. That rules out the explanation you might expect from chapter 3 ("Control+A moved the cursor to the start of the line, so the text was inserted in front"), which would have produced finaldraft.

A failure message is evidence. Write down what it rules out before forming a theory.

Step 2: watch the session#

Every session on TestingBot is recorded. Open the build in the dashboard, then the failed session: the test page shows the video, the logs and the test's metadata. You can also fetch the same data from the REST API, which is handy in CI:

curl -u "$TB_KEY:$TB_SECRET" \
  "https://api.testingbot.com/v1/tests/d0724259dadd-7823b14dafb8-0a82fd9e9bf4-179449731867-34434667"

The JSON includes success, browser, os, a link to the video, and links to the session's logs. The logs differ per browser. For this session there was a Selenium log and a VM log; the Chrome session also had a browser log with the browser console, which is where JavaScript errors show up. Not every browser exposes its console to automation, so when a page error is suspected, reproduce it in a browser that does, or capture errors in the page yourself.

Watching the video of a failed session next to a passing one is often the fastest way to spot the difference: a dialog that only one browser shows, a page that has not finished loading, an element under a cookie banner. See debugging tests for every tool on the test page.

A test fails only in Safari with assert 'draft' == 'final'. Which conclusion does the message itself support?

If the element were missing, the test would fail earlier with NoSuchElementException; if the text were inserted at the start, the value would be finaldraft. draft means nothing after the select-all step arrived.

Step 3: measure inside the browser#

The obvious fix is to use the macOS modifier. Selenium reports the platform in the session's capabilities, so the test can choose:

def select_all_modifier(driver):
    platform = driver.capabilities.get("platformName", "").lower()
    return Keys.COMMAND if platform in ("mac", "macos") else Keys.CONTROL

(Safari on the grid reported platformName as macOS; Chrome on Windows 11 as windows.) We tried box.send_keys(Keys.COMMAND, "a") followed by box.send_keys("final"). Safari still ended with draft.

When a fix that should work does not, stop guessing and measure. A one-line script shows what the browser thinks is selected after the shortcut:

box.send_keys(Keys.COMMAND, "a")
print(driver.execute_script(
    "return [arguments[0].selectionStart, arguments[0].selectionEnd]", box))

It printed [0, 5]: the whole word was selected, so Command+A worked. The problem was the next step. After a modifier chord sent with send_keys, the following send_keys("final") did not change the value in this Safari version. Rewriting the same keystrokes as explicit key-down and key-up actions did:

from selenium.webdriver.common.action_chains import ActionChains

def test_select_all_shortcut(driver):
    driver.get(FORM)
    box = driver.find_element(By.NAME, "my-textarea")
    box.send_keys("draft")

    modifier = select_all_modifier(driver)
    (ActionChains(driver)
        .key_down(modifier).send_keys("a").key_up(modifier)
        .send_keys("final")
        .perform())

    assert box.get_property("value") == "final"

That passed on Chrome and Firefox on Windows 11 and on Safari on macOS Tahoe. The lesson is general: keyboard handling sits in the driver as well as the browser, and each vendor writes its own driver. When keyboard input behaves differently in one browser, the Actions API (which spells out every key press and release) is the most predictable way to send it.

Step 4: ask whether the test needed it#

The ActionChains version is correct, but look at what the test is for. If the point is "the user replaces the text", the shortcut is incidental, and Selenium has a command for that intent:

def test_replace_textarea_text(driver):
    driver.get(FORM)
    box = driver.find_element(By.NAME, "my-textarea")
    box.send_keys("draft")

    box.clear()
    box.send_keys("final")

    assert box.get_property("value") == "final"

That passes everywhere and does not depend on any OS keybinding or driver detail. Keep the shortcut version only if select-all itself is a feature you are testing, for example in a rich text editor.

You have a working Safari fix using ActionChains and the platform modifier. The test is about saving an edited comment. What should the final version use?

Test the user's intent with the most direct command. clear() works in every browser without platform branches. Setting the value with JavaScript skips the input events the app may rely on, and skipping macOS hides the case you just found.

When you need to see it yourself#

Some failures are easier to understand by hand. Two ways to get your hands on the exact browser:

  • Pause a running test. On the test page of a running Selenium session, the interactive session feature lets you pause the script and take over the remote browser with mouse, keyboard and developer tools, then inspect the state at the failing step.
  • Open a live session. Start a live browser with the same browser, version and OS as the failing configuration, and try the steps by hand with the browser's own developer tools. If your site is not public, the TestingBot Tunnel makes it reachable.

A checklist for one-browser failures#

  1. Read the failure. What value or state did you get, and what does that rule out?
  2. Reproduce in one configuration. pytest -k safari or --project=safari-tahoe, so each attempt is one session.
  3. Compare with a passing browser. Same test, video side by side.
  4. Find the layer. Engine (rendering, JavaScript), OS (keys, fonts, controls), driver (how commands are carried out) or the test's own assumptions.
  5. Measure in the browser with execute_script or page.evaluate instead of theorising.
  6. Fix at the right level. An app bug gets an app fix and a test that keeps it fixed; a test habit gets a better test; a driver quirk gets the Actions API or a more direct command.
  7. Run the whole matrix again, because a fix for one browser can break another.

Summary#

  • The failure message narrows the search before you open anything.
  • Video, logs and the REST API show what happened in each session; logs differ per browser.
  • When the obvious fix fails, measure inside the browser.
  • Keyboard input passes through the vendor's driver; the Actions API is the most predictable way to send shortcuts.
  • Prefer commands that express intent, such as clear(), over OS-specific shortcuts.