Skip to main content

XCUITest Intro

TestingBot provides a grid of physical iOS devices in the cloud, ready to run your XCUITests.
XCUITest (a framework within the larger XCTest Testing framework) is Apple's automation framework, which comes with Apple's XCode IDE, capable of running automated UI tests on iPhone and iPad devices.

Setup

XCUITest is packaged in a zip file. You will need two files to perform XCUITesting on our cloud:

  • An .ipa file of the actual mobile application that you want to test.
  • Another .zip file of the XCUITests (the -Runner.app test runner, see Build XCUITest).

XCUITest runs on TestingBot's real iOS devices, so build both for a device (iphoneos), not for the simulator. Your build does not need to be signed for TestingBot's devices: TestingBot re-signs the app and the test runner with its own provisioning profile before installing them on the device. An unsigned build (for example from xcodebuild build-for-testing with CODE_SIGNING_ALLOWED=NO) works, and so does a development or ad hoc signed build.

If you just want to give this a try, we have a TestingBot XCUITest Demo App which you can use with the following steps.

TestingBot CLI

The easiest way to run XCUITests on TestingBot is using our CLI tool. It handles uploading your app, test ZIP, and running tests in a single command.

Installation

npm install -g @testingbot/cli

Authentication

Authenticate using this command:

testingbot login

This opens your browser for authentication. After logging in, your credentials are saved to ~/.testingbot and you can start using the CLI.

Run testingbot whoami to see which account the CLI is using, your plan, minutes left and concurrency, and where the credentials come from (options, TB_KEY/TB_SECRET or ~/.testingbot). testingbot logout deletes ~/.testingbot.

Other Authentication Methods

  • Command-line options: --api-key and --api-secret
  • Environment variables: TB_KEY and TB_SECRET
  • Config file: Create ~/.testingbot with content key:secret

Quick Start

Run XCUITests with a single command:

testingbot xcuitest app.ipa app-test.zip --device "iPhone 16" --real-device

To see the device names and OS versions that --device and --platform-version accept, run testingbot devices --framework xcuitest. Add --real to list real devices with their current availability, or --json for machine-readable output.

The CLI will:

  1. Upload your app IPA to TestingBot
  2. Upload your test ZIP
  3. Start the test run
  4. Show real-time progress and results

Upload iOS App

You can upload your .ipa file to TestingBot storage with our REST-API.

When using the CLI, the app upload is handled automatically as part of the testingbot xcuitest command. Simply specify your app IPA file as the first argument:

testingbot xcuitest /path/to/app/sample-debug.ipa /path/to/test/app-test.zip --device "iPhone 16" --real-device

Alternatively, you can use the --app option:

testingbot xcuitest --app /path/to/app/sample-debug.ipa --test-app /path/to/test/app-test.zip --device "iPhone 16" --real-device
curl -u api_key:api_secret \
-X POST "https://api.testingbot.com/v1/app-automate/xcuitest/app" \
-F "file=@/path/to/app/file/sample-debug.ipa"

The API response will include an id response, which you can use in the other API calls.

{
    "id": 4012
}

Upload XCUITest Suite

If you do not have a .zip file yet, please see our Build Testsuite documentation.

When using the CLI, the test ZIP upload is handled automatically. Specify your test ZIP as the second argument:

testingbot xcuitest app.ipa app-test.zip --device "iPhone 16" --real-device

Or use the --test-app option:

testingbot xcuitest --app app.ipa --test-app app-test.zip --device "iPhone 16" --real-device

To upload the XCUITest suite, in .zip file, please use our REST-API.

curl -u api_key:api_secret \
-X POST "https://api.testingbot.com/v1/app-automate/xcuitest/:id/tests" \
-F "file=@/path/to/app/file/application-debug-test.zip"

Replace the :id with the identifier you received during the app upload call. (In this example 4012)

Run XCUI tests

Once you've uploaded both your app and test suite, you can start a XCUITest on the TestingBot cloud.

Use the CLI to run tests with device selection options:

# Basic usage with device selection
testingbot xcuitest app.ipa app-test.zip --device "iPhone 16" --real-device

# With test name and build identifier
testingbot xcuitest app.ipa app-test.zip \
  --device "iPhone 16" \
  --real-device \
  --name "Smoke Tests" \
  --build "v2.1.0"

Device Options

Option Description
--device <name> Device name (e.g., "iPhone 16", "iPad.*")
--platform-version <version> iOS version (e.g., "17.0", "18.2")
--real-device Use a real device instead of simulator
--tablet-only Only allocate tablet devices (iPads)
--phone-only Only allocate phone devices (iPhones)
--orientation <orientation> Screen orientation: PORTRAIT or LANDSCAPE

Localization & Network

# With localization settings
testingbot xcuitest app.ipa app-test.zip \
  --device "iPhone 16" \
  --real-device \
  --locale "DE" \
  --language "de" \
  --timezone "Europe/Berlin"

# With network throttling and geolocation
testingbot xcuitest app.ipa app-test.zip \
  --device "iPhone 16" \
  --real-device \
  --throttle-network "3G" \
  --geo-country-code "DE"

Specify one or more capabilities, to indicate on which iOS devices you want to run your tests on.

Replace the :id with the identifier you received during the app upload call. (In this example 4012)

We offer special parameters which you can use to allocate a device:

Regex Input Result
"iPhone.*" This will allocate any available iPhone device (phone)
"*" This will allocate a random available device
"iPhone (8|11)" This will allocate either an iPhone 8 or 11
"iPhone 6S?$" This will allocate either an iPhone 6 or 6S

Some Examples:

# find any iPhone, except 6 or 6S
-d '{"capabilities":[{"deviceName":"^iPhone(?! 6S?$).*", "platformName":"iOS", "realDevice": true}]}'

# find any device which name starts with iPad
-d '{"capabilities":[{"deviceName":"iPad.*", "platformName":"iOS", "realDevice": true}]}'

Tablet Only

You can specify the tabletOnly capability when you only want to allocate a tablet device.

Phone Only

You can specify the phoneOnly capability when you only want to allocate a phone device.

-d '{"capabilities":[{"deviceName":"*", "tabletOnly": true, "platformName":"iOS", "realDevice": true}]}'

-d '{"capabilities":[{"deviceName":"*", "phoneOnly": true, "platformName":"iOS", "realDevice": true}]}'

View XCUITest Results

The XCUITest results will be available in the TestingBot dashboard.

Each test result contains a video, test logs and other meta data generated during the test run.

By default, the CLI waits for test completion and displays real-time progress:

testingbot xcuitest app.ipa app-test.zip --device "iPhone 16" --real-device

The CLI will show:

  • Upload progress for app and test ZIP
  • Device allocation status
  • Live output from XCUITests
  • Final pass/fail status

Async Mode

Use --async to start tests without waiting for results:

testingbot xcuitest app.ipa app-test.zip --device "iPhone 16" --real-device --async

Download Reports

Download a JUnit or HTML report after test completion:

testingbot xcuitest app.ipa app-test.zip --device "iPhone 16" --real-device \
  --report junit \
  --report-output-dir ./reports

You can also use an API call to get the results from the run(s):

curl -u api_key:api_secret \
"https://api.testingbot.com/v1/app-automate/xcuitest/:id"

Replace the :id with the identifier you received during the app upload call.

Retry Failed Tests

Pass --retry <count> (0-2) to start a failed XCUITest run again on the app you already uploaded, up to that many times. Only the last attempt counts: it decides the exit code, the --report and the downloaded artifacts.

testingbot xcuitest app.ipa app-test.zip --device "iPhone 16" --real-device --retry 2

--retry cannot be combined with --async, since retrying needs the result of the run.

Timeout

The CLI waits up to 60 minutes for the results. Pass --timeout <minutes> to change that. The clock starts when the tests start, after the uploads, and includes every --retry attempt. When the time is up, the CLI stops the runs that are still queued or running, prints the results so far and fails with exit code 2.

testingbot xcuitest app.ipa app-test.zip --device "iPhone 16" --real-device --timeout 30

Queue Priority

When all your parallel slots are busy, new runs wait for one to free up. Pass --priority high to start these runs before your other waiting runs, for example a release candidate or a hotfix, or --priority low for a nightly run that can wait. Runs with the same priority start oldest first; the default is normal.

testingbot xcuitest app.ipa app-test.zip --device "iPhone 16" --real-device --priority high

Priority only orders your own waiting runs. It does not take slots from other accounts or raise your parallel limit.

Download Artifacts

Use --download-artifacts to download the video, logs and screenshots of each run once the tests have finished. They are saved as one zip, with a folder per run, named after --name (or timestamped). --download-artifacts failed only downloads the runs that failed.

testingbot xcuitest app.ipa app-test.zip --device "iPhone 16" --real-device \
  --download-artifacts failed \
  --artifacts-output-dir ./artifacts

Job Summary and Annotations

In GitHub Actions the CLI adds the results to the job summary, with a link to each run, and annotates every failed run so it shows on the workflow run and the pull request. In Azure Pipelines the same table is attached to the build and failures are logged as errors. Set TESTINGBOT_CI_SUMMARY=0 to turn this off.

CI/CD Metadata

Attach the branch, commit, pull request and repository a run was built from. The run page in the dashboard then links to them on your Git host: GitHub, GitLab, Bitbucket, Azure Repos or a self-hosted server.

Inside GitHub Actions, GitLab CI, Bitbucket Pipelines, Azure Pipelines and CircleCI, the CLI reads the repository, branch, commit and pull request from the CI environment, so no flags are needed. Outside those, or to override what was detected, pass them yourself:

testingbot xcuitest app.ipa app-test.zip --device "iPhone 16" --real-device \
  --branch "feature/login" \
  --commit-sha "4e976b0ce06454ca61bf5d3c8b20ac0fdc457021" \
  --pr-url "https://github.com/acme/app/pull/42" \
  -m team=mobile -m env=staging
Option Description
--branch <name> Git branch this run was built from
--commit-sha <sha> Git commit the run tested
--pull-request-id <id> Pull request (or merge request) number
--pr-url <url> Pull request URL
--repo-owner <owner>, --repo-name <name> A GitHub repository (organization or user, and repository name)
-m, --metadata <KEY=VALUE> Free-form key/value pairs stored with the run (repeatable)

Send a metadata object with the run call. Every field is optional; repoUrl is the repository's web URL on any Git host.

curl -u api_key:api_secret \
-X POST "https://api.testingbot.com/v1/app-automate/xcuitest/:id/run" \
-d '{
  "capabilities": [{ "deviceName": "iPhone 16" }],
  "metadata": {
    "repoUrl": "https://dev.azure.com/acme/mobile/_git/app",
    "branch": "feature/login",
    "commitSha": "4e976b0ce06454ca61bf5d3c8b20ac0fdc457021",
    "pullRequestId": "42",
    "custom": { "team": "mobile" }
  }
}' \
-H "Content-Type: application/json"