Stabilize Screenshots for Visual Testing
During visual testing, we're comparing a screenshot with a previous screenshot. If a difference is detected, the visual test is considered a failed test.
In order to avoid false positives during the comparison of two screenshots, it's important to make sure we prevent as much unwanted and unimportant dynamic data from the screenshot as possible.
Freeze CSS Animations
Right before TestingBot takes a screenshot, we will disable any CSS animations currently running on the page.
TestingBot will freeze CSS animation and transition styles. Once the screenshot has been taken, the CSS animations will run just like before.
Lazy Loaded Images
Some images on the webpage might be marked as lazy loading images. If you are taking a full-page screenshot, TestingBot will force-load all images marked as lazy images, to make sure the entire page is visually tested.
Mobile Statusbars
When testing on (physical) mobile devices, a statusbar might be present on the screen, containing the current time. TestingBot will automatically remove the statusbar from the image, because otherwise the time indicator would trigger a visual difference compared to the baseline.
Date/Time on webpages
A webpage might display the current date or time. In order to avoid marking this as a difference, the date and time of the webpage is frozen to 2024-01-01T12:00:00Z.
Once the screenshot has been taken, the date and time is unfrozen.
Native browser controls
Right before TestingBot takes a screenshot, the caret and the scrollbar on the page will be hidden, in order to avoid false positives with focused text inputs. After the screenshot, these changes will be reverted.
Visual Test Options
TestingBot offers these features to stabilize your screenshots:
Ignore Specific Regions
You can ignore specific regions, defined in pixel coordinates, from the screenshot:
ignoreRegions = [{
"x1": number,
"y1": number,
"x2": number,
"y2": number
}]Ignore Specific Selectors
Pass an array of CSS Selectors to ignore these DOM elements during the visual check.
ignoreSelectors = ["body div.test", "#sidebar"]Only screenshot a specific selector
Pass a CSS selector to take a screenshot of this specific element only.
selector = "body div.test"Scheduled Visual Tests
For visual tests you schedule in the dashboard, two more settings decide whether a run fails. Both are per test and need no code.
Mismatch threshold
A run fails when more than this share of pixels differs from the baseline. New tests start at 1%; you can set anything between 0.1% and 5% on the test's settings page. The slider shows how many of your last ten runs would have passed at the value you pick, so you can size it against your own history rather than guessing.
The comparison is tolerant of anti-aliasing before the threshold is applied: a pixel only counts as changed when a colour channel differs beyond a tolerance and at least one neighbouring pixel changed too. Sub-pixel font jitter and isolated stray pixels do not fail a run.
Ignored regions
Parts of a page that change on their own, a carousel, a clock, a rotating ad, will differ on every run and fail a test for no useful reason. On the test's settings page you can drag a box over those areas of the baseline to ignore them. Ignored regions are painted out of both the screenshot and the baseline before they are compared, so nothing inside one can fail a run. You can copy the regions you drew onto every browser in the test, scaled to each baseline's width.
After a failure, TestingBot also looks at the last three runs of that browser and proposes the regions that changed in every one of them. Nearby changes are merged into one rectangle, so a widget with several moving parts comes back as a single region rather than a box per line. These appear in amber on the settings page, drawn over the latest run, for you to accept or dismiss individually. Accepting one adds it to the ignored regions from the next run onwards. If most of the difference is the same on every run, the page itself has changed and you should approve a run as the new baseline instead.
