Skip to main content

Smart TV Testing

Run tests on physical Smart TV devices with TestingBot, using the open-source test framework Appium.

Currently TestingBot supports running automated tests on physical AppleTV 4k devices.

To get started with testing your Smart TV app in the cloud, you will need to follow these steps:

  • Upload your native Smart TV app, either to a public URL or using TestingBot Storage.
  • Create an Appium test script that will connect to TestingBot.
  • The results will be available in the TestingBot dashboard, together with a video of the test and logs.

Upload Smart TV App

To run Smart TV tests on TestingBot, you will first have to upload your .ipa file to TestingBot Storage.

curl -X POST "https://api.testingbot.com/v1/storage" \
-u key:secret -F "file=@/path/to/app/file/tvos.ipa"

This call will return a unique identifier for your SmartTV app ({"app_url": "tb://..."}), which you can use in the capabilities in your test (see example below).

More information regarding this API call and other similar calls (update uploaded file, delete uploaded file, list uploaded files) are available in the TestingBot API documentation.

Note: uploads to TestingBot Storage are automatically deleted after 60 days.

Run your first SmartTV test

Once you've uploaded your Smart TV app, you can start with running your first test on the TestingBot SmartTV cloud.

TestingBot currently only supports running tests on physical AppleTV 4k devices. Please see the example tests below, where we use the TestingBot tvOS example app to input two numbers in the calculator and verify its sum.

The examples read your TestingBot credentials from the TESTINGBOT_KEY and TESTINGBOT_SECRET environment variables. You can find both in your TestingBot account:

export TESTINGBOT_KEY=your-key
export TESTINGBOT_SECRET=your-secret
require 'appium_lib_core'

caps = {
  "platformName" => "tvOS",
  "appium:automationName" => "XCUITest",
  "appium:deviceName" => "Apple TV 4K",
  "appium:platformVersion" => "17.4",
  "appium:app" => "https://github.com/testingbot/tvos-example/releases/download/1.0.0/tvos-example.ipa",
  "tb:options" => {
    "key" => ENV["TESTINGBOT_KEY"],
    "secret" => ENV["TESTINGBOT_SECRET"],
    "name" => "Ruby tvOS Example",
    "realDevice" => true
  }
}

driver = Appium::Core.for(caps: caps, appium_lib: { server_url: "https://hub.testingbot.com/wd/hub" }).start_driver

begin
  wait = Selenium::WebDriver::Wait.new(timeout: 30)
  input_field1 = wait.until { driver.find_element(:accessibility_id, "inputField1") }
  input_field1.click
  input_field1.send_keys "5"

  # Move the focus to the second field and select it
  driver.execute_script("mobile: pressButton", { name: "down" })
  driver.execute_script("mobile: pressButton", { name: "select" })
  driver.switch_to.active_element.send_keys "10"

  # Move the focus to the result field
  driver.execute_script("mobile: pressButton", { name: "down" })
  driver.execute_script("mobile: pressButton", { name: "select" })

  result = driver.find_element(:accessibility_id, "resultField")
  raise "Expected 15, got #{result.text}" unless result.text == "15"
  puts "Test passed"
ensure
  driver.quit
end
const { remote } = require('webdriverio');

(async () => {
    const driver = await remote({
        protocol: 'https',
        hostname: 'hub.testingbot.com',
        port: 443,
        path: '/wd/hub',
        connectionRetryTimeout: 420000, // real devices can queue for up to ~390s
        connectionRetryCount: 0,        // a retry would queue a second device request
        capabilities: {
            platformName: 'tvOS',
            'appium:automationName': 'XCUITest',
            'appium:deviceName': 'Apple TV 4K',
            'appium:platformVersion': '17.4',
            'appium:app': 'https://github.com/testingbot/tvos-example/releases/download/1.0.0/tvos-example.ipa',
            'tb:options': {
                key: process.env.TESTINGBOT_KEY,
                secret: process.env.TESTINGBOT_SECRET,
                name: 'Node.js tvOS Example',
                realDevice: true
            }
        }
    });

    try {
        const inputField1 = await driver.$('~inputField1');
        await inputField1.waitForDisplayed({ timeout: 30000 });
        await inputField1.click();
        await inputField1.setValue('5');

        // Move the focus to the second field and select it
        await driver.execute('mobile: pressButton', { name: 'down' });
        await driver.execute('mobile: pressButton', { name: 'select' });
        const active = await driver.$(await driver.getActiveElement());
        await active.setValue('10');

        // Move the focus to the result field
        await driver.execute('mobile: pressButton', { name: 'down' });
        await driver.execute('mobile: pressButton', { name: 'select' });

        const resultText = await driver.$('~resultField').getText();
        if (resultText !== '15') {
            throw new Error(`Expected 15, got ${resultText}`);
        }
        console.log('Test passed');
    } finally {
        await driver.deleteSession();
    }
})().catch((err) => {
    console.error(err);
    process.exit(1);
});
import os
from appium import webdriver
from appium.options.common import AppiumOptions
from appium.webdriver.common.appiumby import AppiumBy
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC

options = AppiumOptions().load_capabilities({
    "platformName": "tvOS",
    "appium:automationName": "XCUITest",
    "appium:deviceName": "Apple TV 4K",
    "appium:platformVersion": "17.4",
    "appium:app": "https://github.com/testingbot/tvos-example/releases/download/1.0.0/tvos-example.ipa",
    "tb:options": {
        "key": os.environ["TESTINGBOT_KEY"],
        "secret": os.environ["TESTINGBOT_SECRET"],
        "name": "Python tvOS Example",
        "realDevice": True
    }
})

driver = webdriver.Remote("https://hub.testingbot.com/wd/hub", options=options)
try:
    wait = WebDriverWait(driver, 30)
    input_field1 = wait.until(EC.presence_of_element_located((AppiumBy.ACCESSIBILITY_ID, "inputField1")))
    input_field1.click()
    input_field1.send_keys("5")

    # Move the focus to the second field and select it
    driver.execute_script("mobile: pressButton", {"name": "down"})
    driver.execute_script("mobile: pressButton", {"name": "select"})
    driver.switch_to.active_element.send_keys("10")

    # Move the focus to the result field
    driver.execute_script("mobile: pressButton", {"name": "down"})
    driver.execute_script("mobile: pressButton", {"name": "select"})

    result = driver.find_element(AppiumBy.ACCESSIBILITY_ID, "resultField")
    assert result.text == "15", f"Expected 15, got {result.text}"
finally:
    driver.quit()
import io.appium.java_client.AppiumBy;
import io.appium.java_client.AppiumDriver;
import io.appium.java_client.ios.options.XCUITestOptions;
import org.openqa.selenium.WebElement;
import org.openqa.selenium.support.ui.ExpectedConditions;
import org.openqa.selenium.support.ui.WebDriverWait;

import java.net.URL;
import java.time.Duration;
import java.util.Map;

public class TvOSTest {
    public static void main(String[] args) throws Exception {
        XCUITestOptions options = new XCUITestOptions()
            .setPlatformName("tvOS")
            .setDeviceName("Apple TV 4K")
            .setPlatformVersion("17.4")
            .setApp("https://github.com/testingbot/tvos-example/releases/download/1.0.0/tvos-example.ipa");
        options.setCapability("tb:options", Map.of(
            "key", System.getenv("TESTINGBOT_KEY"),
            "secret", System.getenv("TESTINGBOT_SECRET"),
            "name", "Java tvOS Example",
            "realDevice", true
        ));

        // AppiumDriver keeps platformName "tvOS"
        AppiumDriver driver = new AppiumDriver(new URL("https://hub.testingbot.com/wd/hub"), options);
        try {
            WebDriverWait wait = new WebDriverWait(driver, Duration.ofSeconds(30));
            WebElement inputField1 = wait.until(ExpectedConditions.presenceOfElementLocated(AppiumBy.accessibilityId("inputField1")));
            inputField1.click();
            inputField1.sendKeys("5");

            // Move the focus to the second field and select it
            driver.executeScript("mobile: pressButton", Map.of("name", "down"));
            driver.executeScript("mobile: pressButton", Map.of("name", "select"));
            driver.switchTo().activeElement().sendKeys("10");

            // Move the focus to the result field
            driver.executeScript("mobile: pressButton", Map.of("name", "down"));
            driver.executeScript("mobile: pressButton", Map.of("name", "select"));

            String resultText = driver.findElement(AppiumBy.accessibilityId("resultField")).getText();
            if (!"15".equals(resultText)) {
                throw new AssertionError("Expected 15, got " + resultText);
            }
            System.out.println("Test passed");
        } finally {
            driver.quit();
        }
    }
}
using OpenQA.Selenium;
using OpenQA.Selenium.Appium;
using OpenQA.Selenium.Remote;
using OpenQA.Selenium.Support.UI;
using System;
using System.Collections.Generic;

class TvOSTest
{
    static void Main(string[] args)
    {
        var options = new AppiumOptions
        {
            PlatformName = "tvOS",
            AutomationName = "XCUITest",
            DeviceName = "Apple TV 4K",
            PlatformVersion = "17.4",
            App = "https://github.com/testingbot/tvos-example/releases/download/1.0.0/tvos-example.ipa"
        };
        options.AddAdditionalAppiumOption("tb:options", new Dictionary<string, object>
        {
            ["key"] = Environment.GetEnvironmentVariable("TESTINGBOT_KEY") ?? "",
            ["secret"] = Environment.GetEnvironmentVariable("TESTINGBOT_SECRET") ?? "",
            ["name"] = "C# tvOS Example",
            ["realDevice"] = true
        });

        // RemoteWebDriver keeps platformName "tvOS"; IOSDriver would overwrite it with "iOS".
        // The 420s command timeout covers the time a busy device can stay queued (~390s).
        var driver = new RemoteWebDriver(new Uri("https://hub.testingbot.com/wd/hub"), options.ToCapabilities(), TimeSpan.FromSeconds(420));
        try
        {
            var wait = new WebDriverWait(driver, TimeSpan.FromSeconds(30));
            var inputField1 = wait.Until(d => d.FindElement(MobileBy.AccessibilityId("inputField1")));
            inputField1.Click();
            inputField1.SendKeys("5");

            // Move the focus to the second field and select it
            driver.ExecuteScript("mobile: pressButton", new Dictionary<string, object> { ["name"] = "down" });
            driver.ExecuteScript("mobile: pressButton", new Dictionary<string, object> { ["name"] = "select" });
            driver.SwitchTo().ActiveElement().SendKeys("10");

            // Move the focus to the result field
            driver.ExecuteScript("mobile: pressButton", new Dictionary<string, object> { ["name"] = "down" });
            driver.ExecuteScript("mobile: pressButton", new Dictionary<string, object> { ["name"] = "select" });

            var resultText = driver.FindElement(MobileBy.AccessibilityId("resultField")).Text;
            if (resultText != "15")
            {
                throw new Exception($"Expected 15, got {resultText}");
            }
            Console.WriteLine("Test passed");
        }
        finally
        {
            driver.Quit();
        }
    }
}

View SmartTV Test Results

Once your test has finished running, you will see a new test entry in the TestingBot dashboard.. If you click the test, you will see detailed information about the test, including a video recording of the test, logs and screenshots.

You can also use the TestingBot API to programmatically fetch the test results.