Appium Visual Testing

Setup

TestingBot added a new Javascript Executor command which can be used in Appium tests, called tb:visual.snapshot. By passing a unique identifier with this command, TestingBot will compare the initial screenshot taken with this identifier to the current screenshot.

TestingBot will perform a pixel comparison scan, comparing the screenshot of your mobile app or website with the original golden image (baseline). If TestingBot detects the number of different pixels to be higher than the supplied threshold, the test will be marked as failed.

To add one or more visual checks to your Appium tests, please see the example below, where we save a screenshot of the current mobile app/website state as uniqueIdentifier.

driver.execute_script('tb:visual.snapshot=uniqueIdentifier')

((JavascriptExecutor)driver).executeScript("tb:visual.snapshot=uniqueIdentifier");

$web_driver->executeScript('tb:visual.snapshot=uniqueIdentifier');

driver.execute_script('tb:visual.snapshot=uniqueIdentifier')

driver.executeScript('tb:visual.snapshot=uniqueIdentifier')

((IJavaScriptExecutor)driver).ExecuteScript("tb:visual.snapshot=uniqueIdentifier");

Visual Comparison Response

During your Appium test, you can perform one or more visual comparisons and receive back a response in less than 2 seconds. For each visual test, you will receive back a response indicating:

• Does the screenshot match the baseline?
• What are the number of pixels that are different between the two snapshots?

{
    	"match": false,
    	"pixelDifference": 341575
}

Please see the example below on how to check whether a visual change was detected:

visual_response = driver.execute_script('tb:visual.snapshot=uniqueIdentifier')
visual_test_passed = visual_response["match"] == true

String jsonString = (String) ((JavascriptExecutor) driver).executeScript("tb:visual.snapshot=uniqueIdentifier");

ObjectMapper objectMapper = new ObjectMapper();
try {
    JsonNode jsonNode = objectMapper.readTree(jsonString);
    Boolean visualTestPassed = jsonNode.get("match").asBoolean();
} catch (Exception e) {
    e.printStackTrace();
}

$visual_response = $web_driver->executeScript('tb:visual.snapshot=uniqueIdentifier');
$visual_json = json_decode($visual_response, true);
$visual_test_passed = $visual_json["match"] === true;

json_string = driver.execute_script("tb:visual.snapshot=uniqueIdentifier")

# Decode the JSON response using the json module
json_object = json.loads(json_string)

# Check if the 'matched' key is true
visusal_test_passed = 'matched' in json_object and json_object['matched']

const visual_response = driver.executeScript('tb:visual.snapshot=uniqueIdentifier')
const json_response = JSON.parse(visual_response)
const visual_test_passed = json_response['match'] === true

var jsonScript = "tb:visual.snapshot=uniqueIdentifier";
var jsonResult = (string)driver.ExecuteScript(jsonScript);

var jsonObject = JsonConvert.DeserializeObject<JsonObject>(jsonResult);
var visualTestPassed = jsonObject != null && jsonObject.Matched

Visual Baseline

The first time that you run the compare command with a given identifier, TestingBot will automatically mark the screenshot as the baseline. If you want to update the baseline after it has already been saved, you can use the tb:visual.baseline command to take a new screenshot and set it as the new baseline image.

Please see the example below on how to update the baseline for a given identifier:

driver.execute_script('tb:visual.baseline=uniqueIdentifier')

((JavascriptExecutor) driver).executeScript("tb:visual.baseline=uniqueIdentifier");

$web_driver->executeScript('tb:visual.baseline=uniqueIdentifier');

driver.execute_script("tb:visual.baseline=uniqueIdentifier")

driver.executeScript('tb:visual.baseline=uniqueIdentifier')

driver.ExecuteScript("tb:visual.baseline=uniqueIdentifier");

Visual Testing Options

TestingBot offers various options to customize the visual comparison tests. Options can be passed with the tb:visual.snapshot command:

const visualSnapshotJsonCmd = JSON.stringify({
  name: 'testing123',
  options: {
    threshold: 0.3
  }
})

driver.executeScript(`tb:visual.baseline=${visualSnapshotJsonCmd}`)

Option Name	Default Option Value	Description
threshold	0.1	This defines the color difference threshold (from 0 to 1). The smaller the number, the more precise the comparison will be.
antialiasing	true	When this is set to true, TestingBot will not count antialiased pixels to the diff of the snapshot.
ignoreRegions	[]	Regions to ignore, defined in pixel coordinates. Needs to be an array with these objects: [{ "x1": number, "y1": number, "x2": number, "y2": number }]
ignoreSelectors Web Testing Only	[]	Pass an array of CSS Selectors to ignore these DOM elements during the visual check. ["body div.test", "#sidebar"]
selector Web Testing Only	-	Pass a CSS selector to take a screenshot of this specific element only. If this is not specified, TestingBot will take a screenshot of the viewport. ["body div.test", "#sidebar"]

visual_snapshot_json_cmd = {
  name: 'testing123',
  options: {
    threshold: 0.3
  }
}.to_json

driver.execute_script("tb:visual.baseline=#{visual_snapshot_json_cmd}")

String visualSnapshotJsonCmd = "{\"name\": \"testing123\", \"options\": {\"threshold\": 0.3}}";
((JavascriptExecutor) driver).executeScript("tb:visual.baseline=" + visualSnapshotJsonCmd);

$visualSnapshotJsonCmd = json_encode([
    'name' => 'testing123',
    'options' => [
        'threshold' => 0.3
    ]
]);

$driver->executeScript("tb:visual.baseline = '{$visualSnapshotJsonCmd}';");

visual_snapshot_json_cmd = json.dumps({
    "name": "testing123",
    "options": {
        "threshold": 0.3
    }
})

driver.execute_script(f"tb:visual.baseline={visual_snapshot_json_cmd};")

const visualSnapshotJsonCmd = JSON.stringify({
  name: 'testing123',
  options: {
    threshold: 0.3
  }
})

driver.executeScript(`tb:visual.baseline=${visualSnapshotJsonCmd}`)

var visualSnapshotJsonCmd = "{\"name\": \"testing123\", \"options\": {\"threshold\": 0.3}}";
((IJavaScriptExecutor)driver).ExecuteScript($"tb:visual.baseline={visualSnapshotJsonCmd};");