Selenium Visual Testing

Setup

TestingBot introduces a new Javascript Executor command which you can use in your Selenium tests, called tb:visual.snapshot. Simply pass a unique identifier to this function and TestingBot will compare the current UI of the webpage with a previous baseline/golden image.

When you run this command for the first time, a screenshot will be taken from the page and will be marked as the baseline. Any future calls with the same identifier will trigger a pixel comparison on TestingBot's end. If TestingBot detects the number of different pixels to be higher than the supplied threshold, the call will return a response indicating the UI has changed.

To add the command to your Selenium tests, please see the example below, where we save a screenshot of the current page as uniqueIdentifier.

driver.execute_script('tb:visual.snapshot=uniqueIdentifier')

((JavascriptExecutor)driver).executeScript("tb:visual.snapshot=uniqueIdentifier");

$web_driver->executeScript('tb:visual.snapshot=uniqueIdentifier');

driver.execute_script('tb:visual.snapshot=uniqueIdentifier')

driver.executeScript('tb:visual.snapshot=uniqueIdentifier')

((IJavaScriptExecutor)driver).ExecuteScript("tb:visual.snapshot=uniqueIdentifier");

The identifier passed to tb:visual.snapshot is unique per browser vendor, not per browser version. TestingBot considers browser versions to generate identical UI's, whereas different browser vendors might generate slightly different UI's (for example Safari vs Chrome).

Visual Comparison Response

Whenever you run a visual comparison during a Selenium test, you will receive back a response in less than 2 seconds. This response indicates whether TestingBot considers the current screenshot and the baseline screenshot to be identical, along with the count of detected differing pixels.

{
    	"match": false,
    	"pixelDifference": 341575
}

Please see the example below on how to check whether a visual change was detected:

visual_response = driver.execute_script('tb:visual.snapshot=uniqueIdentifier')
visual_test_passed = visual_response["match"] == true

String jsonString = (String) ((JavascriptExecutor) driver).executeScript("tb:visual.snapshot=uniqueIdentifier");

ObjectMapper objectMapper = new ObjectMapper();
try {
    JsonNode jsonNode = objectMapper.readTree(jsonString);
    Boolean visualTestPassed = jsonNode.get("match").asBoolean();
} catch (Exception e) {
    e.printStackTrace();
}

$visual_response = $web_driver->executeScript('tb:visual.snapshot=uniqueIdentifier');
$visual_json = json_decode($visual_response, true);
$visual_test_passed = $visual_json["match"] === true;

json_string = driver.execute_script("tb:visual.snapshot=uniqueIdentifier")

# Decode the JSON response using the json module
json_object = json.loads(json_string)

# Check if the 'matched' key is true
visual_test_passed = 'matched' in json_object and json_object['matched']

const visual_response = driver.executeScript('tb:visual.snapshot=uniqueIdentifier')
const json_response = JSON.parse(visual_response)
const visual_test_passed = json_response['match'] === true

var jsonScript = "tb:visual.snapshot=uniqueIdentifier";
var jsonResult = (string)driver.ExecuteScript(jsonScript);

var jsonObject = JsonConvert.DeserializeObject<JsonObject>(jsonResult);
var visualTestPassed = jsonObject != null && jsonObject.Matched

Visual Baseline

The first time you'll run the compare command, TestingBot will automatically mark the screenshot as the baseline. If you want to change the baseline after the first run, you can use the tb:visual.baseline command to take a new screenshot and set it as the baseline (golden image).

You might want to use it to update the baseline when your webpage has changed its content or layout.

Please see the example below on how to set the baseline to a new screenshot:

driver.execute_script('tb:visual.baseline=uniqueIdentifier')

((JavascriptExecutor) driver).executeScript("tb:visual.baseline=uniqueIdentifier");

$web_driver->executeScript('tb:visual.baseline=uniqueIdentifier');

driver.execute_script("tb:visual.baseline=uniqueIdentifier")

driver.executeScript('tb:visual.baseline=uniqueIdentifier')

driver.ExecuteScript("tb:visual.baseline=uniqueIdentifier");

Visual Testing Options

TestingBot provides various options to customize the visual comparison tests. Options can be passed with the tb:visual.snapshot command, for example:

const visualSnapshotJsonCmd = JSON.stringify({
  name: 'testing123',
  options: {
    threshold: 0.3
  }
})

driver.executeScript(`tb:visual.baseline=${visualSnapshotJsonCmd}`)

Option Name	Default Option Value	Description
threshold	0.1	This defines the color difference threshold (from 0 to 1). The smaller the number, the more precise the comparison will be.
antialiasing	true	When this is set to true, TestingBot will not count antialiased pixels to the diff of the snapshot.
ignoreRegions	[]	Regions to ignore, defined in pixel coordinates. Needs to be an array with these objects: [{ "x1": number, "y1": number, "x2": number, "y2": number }]
ignoreSelectors	[]	Pass an array of CSS Selectors to ignore these DOM elements during the visual check. ["body div.test", "#sidebar"]
selector	-	Pass a CSS selector to take a screenshot of this specific element only. If this is not specified, TestingBot will take a screenshot of the viewport. ["body div.test", "#sidebar"]
fullpage	false	If you'd like to take a screenshot of the entire page, specify true. TestingBot will capture the entire page, instead of only the viewport. This feature is currently only available for these browsers: Chrome, Microsoft Edge and Firefox.

visual_snapshot_json_cmd = {
  name: 'testing123',
  options: {
    threshold: 0.3
  }
}.to_json

driver.execute_script("tb:visual.baseline=#{visual_snapshot_json_cmd}")

String visualSnapshotJsonCmd = "{\"name\": \"testing123\", \"options\": {\"threshold\": 0.3}}";
((JavascriptExecutor) driver).executeScript("tb:visual.baseline=" + visualSnapshotJsonCmd);

$visualSnapshotJsonCmd = json_encode([
    'name' => 'testing123',
    'options' => [
        'threshold' => 0.3
    ]
]);

$driver->executeScript("tb:visual.baseline = '{$visualSnapshotJsonCmd}';");

visual_snapshot_json_cmd = json.dumps({
    "name": "testing123",
    "options": {
        "threshold": 0.3
    }
})

driver.execute_script(f"tb:visual.baseline={visual_snapshot_json_cmd};")

const visualSnapshotJsonCmd = JSON.stringify({
  name: 'testing123',
  options: {
    threshold: 0.3
  }
})

driver.executeScript(`tb:visual.baseline=${visualSnapshotJsonCmd}`)

var visualSnapshotJsonCmd = "{\"name\": \"testing123\", \"options\": {\"threshold\": 0.3}}";
((IJavaScriptExecutor)driver).ExecuteScript($"tb:visual.baseline={visualSnapshotJsonCmd};");