Skip to main content

Find elements by image

Some screens have no element tree that Appium can query: games, maps, charts, video players and other views drawn on a canvas or with a custom rendering engine. The Appium Images plugin adds an -image locator strategy, which finds an element by matching a small reference picture (a template) against a screenshot of the device.

Enable the Images plugin

The Images plugin is pre-installed on TestingBot, for real devices as well as for simulators and emulators. It is off by default: switch it on per test with the tb:appiumPlugins capability.

  • Plugins need Appium 2 or Appium 3. They are not available with the legacy Appium 1.
  • The plugins you can enable are images, execute-driver and universal-xml. Other names in the list are ignored.
capabilities = {
  'platformName' => 'Android',
  'appium:automationName' => 'UiAutomator2',
  'appium:deviceName' => 'Pixel 9',
  'appium:app' => 'https://testingbot.com/appium/sample.apk',
  'tb:appiumPlugins' => ['images'],
  'tb:options' => {
    'key' => 'api_key',
    'secret' => 'api_secret',
    'realDevice' => true
  }
}
from appium.options.android import UiAutomator2Options

options = UiAutomator2Options()
options.platform_name = "Android"
options.device_name = "Pixel 9"
options.app = "https://testingbot.com/appium/sample.apk"
options.set_capability("tb:appiumPlugins", ["images"])
options.set_capability("tb:options", {
    "key": "api_key",
    "secret": "api_secret",
    "realDevice": True
})
UiAutomator2Options options = new UiAutomator2Options()
    .setPlatformName("Android")
    .setDeviceName("Pixel 9")
    .setApp("https://testingbot.com/appium/sample.apk");
options.setCapability("tb:appiumPlugins", List.of("images"));

Map<String, Object> tbOptions = new HashMap<>();
tbOptions.put("key", "api_key");
tbOptions.put("secret", "api_secret");
tbOptions.put("realDevice", true);
options.setCapability("tb:options", tbOptions);
const capabilities = {
  platformName: 'Android',
  'appium:automationName': 'UiAutomator2',
  'appium:deviceName': 'Pixel 9',
  'appium:app': 'https://testingbot.com/appium/sample.apk',
  'tb:appiumPlugins': ['images'],
  'tb:options': {
    key: 'api_key',
    secret: 'api_secret',
    realDevice: true
  }
};
var options = new AppiumOptions();
options.PlatformName = "Android";
options.AutomationName = "UiAutomator2";
options.DeviceName = "Pixel 9";
options.App = "https://testingbot.com/appium/sample.apk";
options.AddAdditionalAppiumOption("tb:appiumPlugins", new[] { "images" });
options.AddAdditionalAppiumOption("tb:options", new Dictionary<string, object>
{
    ["key"] = "api_key",
    ["secret"] = "api_secret",
    ["realDevice"] = true
});

Find and tap an element by image

The -image strategy takes the template as a base64-encoded PNG or JPEG. Appium takes a screenshot, looks for the template in it and returns an element at the position of the best match. You can click it like any other element, and read its location and size. If nothing on the screen matches well enough, the lookup fails with the usual "no such element" error.

require 'base64'

template = Base64.strict_encode64(File.binread('images/play-button.png'))
play_button = driver.find_element(:image, template)
play_button.click
import base64
from appium.webdriver.common.appiumby import AppiumBy

with open("images/play-button.png", "rb") as f:
    template = base64.b64encode(f.read()).decode("utf-8")

play_button = driver.find_element(AppiumBy.IMAGE, template)
play_button.click()
import io.appium.java_client.AppiumBy;
import org.openqa.selenium.WebElement;
import java.nio.file.Files;
import java.nio.file.Path;
import java.util.Base64;

String template = Base64.getEncoder()
    .encodeToString(Files.readAllBytes(Path.of("images/play-button.png")));

WebElement playButton = driver.findElement(AppiumBy.image(template));
playButton.click();
// WebdriverIO: a selector ending in .png, .jpg, .jpeg, .gif, .bmp or .svg
// is read from disk, base64-encoded and sent with the -image strategy
const playButton = await driver.$('./images/play-button.png');
await playButton.click();
using System;
using System.IO;
using OpenQA.Selenium.Appium;

var template = Convert.ToBase64String(File.ReadAllBytes("images/play-button.png"));

var playButton = driver.FindElement(new ByImage(template));
playButton.Click();

Tune the image matching

The matching is controlled with Appium settings, which you change during the test through the Settings API. The ones you are most likely to need:

Setting Default What it does
imageMatchThreshold 0.4 The minimum match score, from 0 to 1. Lower it when a visible element is not found; raise it when the wrong element is matched.
fixImageTemplateSize false Shrink the template when it is larger than the screenshot, instead of failing.
defaultImageTemplateScale 1.0 Scale factor applied to every template. Useful when you store templates smaller than they appear on the screen.
fixImageFindScreenshotDims true Resize the screenshot to the screen size, so that a match is tapped at the right coordinates on high-density screens.
getMatchedImageResult false Return the matched image as base64 in the element's visual attribute. Handy when you debug why the wrong spot was matched.

The full list is in the Images plugin documentation.

driver.update_settings('imageMatchThreshold' => 0.6, 'getMatchedImageResult' => true)
driver.update_settings({"imageMatchThreshold": 0.6, "getMatchedImageResult": True})
// AndroidDriver and IOSDriver both implement HasSettings
driver.setSetting("imageMatchThreshold", 0.6);
driver.setSetting("getMatchedImageResult", true);
await driver.updateSettings({ imageMatchThreshold: 0.6, getMatchedImageResult: true });
driver.SetSetting("imageMatchThreshold", 0.6);
driver.SetSetting("getMatchedImageResult", true);

When to use it, and tips

  • Use it for what Appium cannot see: canvas and OpenGL views, game engines, maps, charts, and custom-drawn controls. For standard buttons and text fields, an accessibility id or other regular locator is faster and less brittle.
  • Crop templates from a screenshot of the same device model. A template made on a different screen size or density may no longer match. Keep the crop tight around the element, without background that changes.
  • Every image lookup takes a new screenshot, so it is slower than a regular find. Wait for the screen to settle (for example after an animation) before you look for the image.
  • Don't depend on colours or text that change, such as a counter or a highlighted state. Pick a part of the element that always looks the same.
Was this page helpful?
Last updated