Skip to main content

Appium Visual Testing

Appium is a mobile test automation framework, built to run web and mobile app tests on mobile simulators and physical devices. Visual testing is currently not yet supported by Appium. TestingBot has built a visual regression testing solution for Appium, which you can use without installing any plugins or custom code.

By adding one or more visual checks to your Appium test, you can quickly add visual checkpoints to your Appium test scripts. TestingBot will let you know if a snapshot of your current app looks identical to the baseline that you've set.

Setup

TestingBot added a new Javascript Executor command which can be used in Appium tests, called tb:visual.snapshot. By passing a unique identifier with this command, TestingBot will compare the initial screenshot taken with this identifier to the current screenshot.

TestingBot will perform a pixel comparison scan, comparing the screenshot of your mobile app or website with the original golden image (baseline). A pixel counts as different when its colour differs by more than the threshold; the snapshot only matches the baseline when no pixel differs.

To add one or more visual checks to your Appium tests, please see the example below, where we save a screenshot of the current mobile app/website state as uniqueIdentifier.

driver.execute_script('tb:visual.snapshot=uniqueIdentifier')
((JavascriptExecutor)driver).executeScript("tb:visual.snapshot=uniqueIdentifier");
$web_driver->executeScript('tb:visual.snapshot=uniqueIdentifier');
driver.execute_script('tb:visual.snapshot=uniqueIdentifier')
await driver.executeScript('tb:visual.snapshot=uniqueIdentifier')
((IJavaScriptExecutor)driver).ExecuteScript("tb:visual.snapshot=uniqueIdentifier");

Visual Comparison Response

During your Appium test, you can perform one or more visual comparisons. TestingBot takes the screenshot, compares it and then answers the command. For each visual test, you will receive back a response indicating:

  • Does the screenshot match the baseline?
  • What are the number of pixels that are different between the two snapshots?
  • The URL to the result in the TestingBot dashboard
  • The visualId and runId of the visual test
{
      "match": false,
      "pixelDifference": 341575,
      "url": "https://testingbot.com/members/visual/:visualId/runs/:runId",
      "visualId": ":visualId",
      "runId": ":runId"
}

Please see the example below on how to check whether a visual change was detected:

result = driver.execute_script('tb:visual.snapshot=uniqueIdentifier')
# result is a Hash, for example {"match" => true, "pixelDifference" => 0, ...}
visual_test_passed = result['match'] == true
// import java.util.Map;
@SuppressWarnings("unchecked")
Map<String, Object> result = (Map<String, Object>) ((JavascriptExecutor) driver).executeScript("tb:visual.snapshot=uniqueIdentifier");
boolean visualTestPassed = Boolean.TRUE.equals(result.get("match"));
$result = $web_driver->executeScript('tb:visual.snapshot=uniqueIdentifier');
// $result is an associative array, for example ['match' => true, 'pixelDifference' => 0, ...]
$visual_test_passed = ($result['match'] ?? false) === true;
result = driver.execute_script("tb:visual.snapshot=uniqueIdentifier")

# result is a dict, for example {'match': True, 'pixelDifference': 0, ...}
visual_test_passed = result.get("match", False)
const result = await driver.executeScript('tb:visual.snapshot=uniqueIdentifier')
// result is an object, for example { match: true, pixelDifference: 0, ... }
const visualTestPassed = result.match === true
// using System.Collections.Generic;
var result = (Dictionary<string, object>)((IJavaScriptExecutor)driver).ExecuteScript("tb:visual.snapshot=uniqueIdentifier");
var visualTestPassed = result.TryGetValue("match", out var match) && match is bool b && b;

When a snapshot fails

If TestingBot could not take or compare the screenshot, the command does not throw. It returns match: false together with an error field that says what went wrong. Check for error before you treat match: false as a visual change. The visual testing troubleshooting guide lists the errors and what to do about them.

A snapshot name may contain letters, digits, ., _ and -, up to 128 characters, and must not contain ... Other names are refused with an error.

How long a snapshot takes

A visual check usually returns within a few seconds. When the comparison service does not answer, TestingBot retries it up to four times before it gives up. Make sure the command (read) timeout of your Appium client allows at least two minutes for this call.

Visual Baseline

The first time that you run the compare command with a given identifier, TestingBot will automatically mark the screenshot as the baseline. If you want to update the baseline after it has already been saved, you can use the tb:visual.baseline command to take a new screenshot and set it as the new baseline image.

Please see the example below on how to update the baseline for a given identifier:

driver.execute_script('tb:visual.baseline=uniqueIdentifier')
((JavascriptExecutor) driver).executeScript("tb:visual.baseline=uniqueIdentifier");
$web_driver->executeScript('tb:visual.baseline=uniqueIdentifier');
driver.execute_script("tb:visual.baseline=uniqueIdentifier")
await driver.executeScript('tb:visual.baseline=uniqueIdentifier')
driver.ExecuteScript("tb:visual.baseline=uniqueIdentifier");

Visual Testing Options

TestingBot offers various options to customize the visual comparison tests. Options can be passed with the tb:visual.snapshot command:

const visualSnapshotJsonCmd = JSON.stringify({
  name: 'testing123',
  options: {
    threshold: 0.3
  }
})

driver.executeScript(`tb:visual.snapshot=${visualSnapshotJsonCmd}`)
Option Name Default Option Value Description
threshold 0.1

This defines the color difference threshold (from 0 to 1): how far a pixel's colour may differ from the baseline before it counts as a different pixel. The smaller the number, the more precise the comparison will be. Anti-aliased pixels are never counted as different.

ignoreRegions []

Regions to ignore, defined in pixel coordinates. Needs to be an array with these objects:

[{
  "x1": number,
  "y1": number,
  "x2": number,
  "y2": number
}]
ignoreSelectors

Web Testing Only
[]

Pass an array of CSS Selectors to ignore these DOM elements during the visual check.

["body div.test", "#sidebar"]
selector

Web Testing Only
-

Pass a CSS selector to take a screenshot of this specific element only. If this is not specified, TestingBot will take a screenshot of the viewport.

"body div.test"
visual_snapshot_json_cmd = {
  name: 'testing123',
  options: {
    threshold: 0.3
  }
}.to_json

driver.execute_script("tb:visual.snapshot=#{visual_snapshot_json_cmd}")
String visualSnapshotJsonCmd = "{\"name\": \"testing123\", \"options\": {\"threshold\": 0.3}}";
((JavascriptExecutor) driver).executeScript("tb:visual.snapshot=" + visualSnapshotJsonCmd);
$visualSnapshotJsonCmd = json_encode([
    'name' => 'testing123',
    'options' => [
        'threshold' => 0.3
    ]
]);

$driver->executeScript("tb:visual.snapshot={$visualSnapshotJsonCmd}");
import json

visual_snapshot_json_cmd = json.dumps({
    "name": "testing123",
    "options": {
        "threshold": 0.3
    }
})

driver.execute_script(f"tb:visual.snapshot={visual_snapshot_json_cmd}")
const visualSnapshotJsonCmd = JSON.stringify({
  name: 'testing123',
  options: {
    threshold: 0.3
  }
})

await driver.executeScript(`tb:visual.snapshot=${visualSnapshotJsonCmd}`)
var visualSnapshotJsonCmd = "{\"name\": \"testing123\", \"options\": {\"threshold\": 0.3}}";
((IJavaScriptExecutor)driver).ExecuteScript($"tb:visual.snapshot={visualSnapshotJsonCmd}");
Was this page helpful?
Last updated