Visual Testing Troubleshooting
This page lists the problems we see most often with visual tests, the messages TestingBot shows for them, and what to do about each one.
Automated snapshots
When an automated snapshot (tb:visual.snapshot or visual.snapshot) cannot be taken or compared, the command does not throw. It returns match: false with an error field. Always check for error first: a match: false without one is a real visual difference.
Invalid visual name
Snapshot names may only contain letters, digits, ., _ and -, up to 128 characters, without ... Replace spaces and slashes, for example checkout/step 2 becomes checkout_step-2.
A full-page snapshot only shows the viewport
- The option is called
fullPage, with a capital P. Any other spelling, such asfullpage, is ignored. - Full-page screenshots are available in Chrome, Microsoft Edge and Firefox. Other browsers return a screenshot of the viewport.
- Images and content that load while you scroll are not loaded for you. Scroll through the page in your test before the snapshot.
Element not found
The CSS selector passed in the selector option did not match an element on the page. Check the selector in the browser's developer tools, and wait for the element in your test before you take the snapshot.
Could not capture a screenshot or Empty screenshot
The browser did not return a screenshot. The text after Could not capture a screenshot: is the browser's own reason. Make sure no alert or dialog is open and that the window your test uses is still open when you take the snapshot.
The screenshot exceeded a size limit
Very long pages can produce a full-page screenshot that is too large to process. Take a snapshot of the part you care about with the selector option, or take several snapshots of the viewport at different scroll positions.
Failed to compare after 4 attempts
The screenshot was taken, but the comparison service did not answer, even after retries. Run the test again; if it keeps happening, contact us with the session id.
Playwright or Puppeteer: no result, or Could not reach the visual service
- If
page.evaluatereturns nothing at all, the command was not recognised. Itsargumentsmust be an object, not a JSON string, and the whole command must be valid JSON. See the Playwright and Puppeteer examples. -
Could not reach the visual servicemeans TestingBot gave up waiting after 90 seconds, or could not pass on the command. Run the test again. -
The visual service returned no verdict.means the comparison ended without a result. Run the test again; if it keeps happening, contact us.
My test times out on the snapshot command
A snapshot usually takes a few seconds, but a large screenshot or a retried comparison can take much longer. Allow at least two minutes for the visual command in the command (read) timeout of your Selenium or Appium client. For Playwright and Puppeteer, TestingBot answers within 90 seconds, so keep any timeout you set on page.evaluate above that.
Runs that fail without a real change
Almost every pixel differs
The screenshot and the baseline probably have different dimensions. Use the same screen resolution and device as when the baseline was taken. The run page warns when the sizes differ. If the new size is intended, approve the run with Make this the new baseline ticked, or set a new baseline with tb:visual.baseline.
A few pixels differ on every run
- An automated snapshot matches only when no pixel differs. Raise the
thresholdoption to tolerate small colour differences, or comparepixelDifferencewith a limit of your own. - Content that changes by itself, such as dates, counters, carousels or ads, needs to be left out. Use
ignoreSelectorsorignoreRegionsfor automated snapshots, and ignored regions for scheduled tests. - TestingBot only freezes
Datein the moment of the screenshot. A date the page rendered when it loaded is not changed, see Date/Time on webpages.
I approved a run, but the next run fails again
Approving a run keeps the old baseline. To compare future runs with the new screenshot, approve it with Make this the new baseline ticked. See approve or reject a change.
Scheduled visual tests
When a run of a scheduled visual test stops before the screenshot, the run is marked as failed with the reason, and the test's settings page shows the last failure per browser.
A step failed
Messages that start with Step N: come from the steps before the screenshot. Nothing was compared for that run.
| Message | What to do |
|---|---|
`selector` not found after 15 s |
No visible element matched the selector in time. Check the selector, and add a Wait for an element step if the page needs longer to show it. Elements that are hidden or have no size do not count as found. |
`selector` is not a valid CSS selector |
The browser could not parse the selector. Steps use CSS selectors, not XPath. |
could not click, could not type into, could not hover over, could not scroll to
|
The element was found but did not accept the action. Point the selector at the element itself, for example the input rather than its wrapper. |
out of time, all steps together may take 120 s |
Shorten fixed waits, or replace them with Wait for an element, which continues as soon as the element shows. |
… is not on … |
A Go to a URL step may only open pages on the same host as the test's URL. |
the page did not load again after setting cookie |
After setting cookies the test's URL is loaded again, and that load failed. Check that the page loads with the cookie set. |
Not captured: … points at a private or unreachable host and no tunnel is running
The test's URL is on a private network, or its host no longer resolves. To test an internal or local site, a TestingBot Tunnel has to be running on your account at the moment the test runs. Otherwise, use a URL that is reachable from the internet.
Alerts
- On a free account, alerts for visual tests are sent at most once every 24 hours.
- A webhook URL must be a public
httporhttpsaddress. The request is sent once, with a 25 second timeout, and is not retried; redirects are not followed. - Runs of a CI preview deployment do not trigger the test's alerts; their result is reported on the CI check.
See Alerting for the alert types and the webhook payload.