Exit course
Chapter 8 of 10 Beginner 7 min read 3 quick checks Updated

Run Maestro Flows on TestingBot

On this page
  1. Install and authenticate the TestingBot CLI
  2. Find a device
  3. Your first cloud run
  4. Pin the Maestro version
  5. Parallel runs and shards
  6. Choosing devices without waiting
  7. Variables, secrets and tags
  8. Retries, reports and artifacts
  9. Apps that talk to a private backend
  10. Other run options

Locally you have one emulator and, on a Mac, a few simulators. A real suite needs more: physical iPhones and Android phones, several OS versions, and enough devices to run flows in parallel. This chapter runs the workspace from chapter 6 on TestingBot's devices without changing a single flow file. Everything here comes from the TestingBot Maestro documentation; check it for options added after this course was written.

Install and authenticate the TestingBot CLI#

The TestingBot CLI is an npm package. It uploads your app, zips and uploads your flows, starts the run and streams results back:

npm install -g @testingbot/cli
testingbot login      # opens a browser, stores credentials in ~/.testingbot
testingbot whoami     # shows the account, plan, minutes left and concurrency

On a CI runner, skip the browser login and set TB_KEY and TB_SECRET as environment variables instead (or pass --api-key and --api-secret).

Find a device#

Device names and OS versions must match what the grid offers. List them:

testingbot devices --framework maestro          # emulators and simulators
testingbot devices --framework maestro --real   # real devices and their availability

Your first cloud run#

Upload the Android sample app and run the calculator flows on an emulator. The flows use appId: ${APP_ID} from chapter 4, so pass the Android id:

testingbot maestro sample.apk .maestro \
  --device "Pixel 8" --deviceVersion "14" \
  -e APP_ID=com.sampleapp

The same flows against the iOS simulator build:

testingbot maestro sample.zip .maestro \
  --platform iOS --device "iPhone 17" --deviceVersion "26.0" \
  -e APP_ID=org.reactjs.native.example.SampleApp

And on physical hardware, add --real-device. With an .ipa, real devices are selected automatically:

testingbot maestro app.apk .maestro --device "Galaxy S24" --real-device
testingbot maestro app.ipa .maestro --device "iPhone 17" --real-device

The app formats follow the platform: .apk for Android, .app or a zipped .app for the iOS Simulator, and .ipa for physical iPhones and iPads. The CLI shows upload progress, device allocation and live flow output, then exits with 0 (all flows passed), 2 (flows failed) or 1 (CLI or infrastructure error). That distinction matters in CI, as the next chapter shows.

The flows directory is zipped as-is. If it contains a config.yaml at the top level, it is used just as it is locally: flows globs, tags and env all apply. Make sure every subflow and script a flow references is inside the uploaded folder.

You have sample.zip, the iOS Simulator build, and want to run on a physical iPhone. What do you need?

Simulator builds cannot be installed on hardware. Physical iPhones and iPads take a signed .ipa.

Pin the Maestro version#

Unless you pin one, TestingBot runs flows with a known-good default version (2.6.0 at the time of writing), not necessarily the newest release. Pin the version closest to the one you develop with, so local and cloud runs behave the same. Patch releases are not all on the pinnable list: at the time of writing 2.12.0 is the newest you can pin, so with 2.12.1 installed locally you pin 2.12.0:

testingbot maestro sample.apk .maestro --device "Pixel 8" --maestro-version "2.12.0"

The options page lists which versions can be pinned. A few are pre-installed on the devices; any other version is downloaded when the session starts, which adds to start-up time.

A flow passes locally on Maestro 2.12.1 but fails on TestingBot with an error about a command. You passed no version option. What is the first thing to check?

Without a pin the grid uses its known-good default version, which may be older than yours. Pinning makes local and cloud behave the same.

Parallel runs and shards#

By default each flow runs in its own device session, in parallel, up to your plan's concurrency; anything beyond that is queued until a slot frees up. This is fast, and it is also why chapter 6 insisted that flows must not depend on each other.

To group flows instead, use --shard-split:

# Split the suite across 3 parallel sessions
testingbot maestro sample.apk .maestro --device "Pixel 8" --shard-split 3

# Run every flow in one session, one after another
testingbot maestro sample.apk .maestro --device "Pixel 8" --shard-split 1

Shards are balanced by expected duration, based on earlier runs of each flow, so they finish at about the same time. Shards run on separate devices and never share state.

Choosing devices without waiting#

A specific physical device can be busy. Each session then waits for it, plus its clean-up cycle after the previous test. The device name and version accept patterns, which let the grid pick any free match:

Value Matches
--device ".*Galaxy.*" any available Galaxy phone or tablet
--device "Pixel.*" any available Pixel
--deviceVersion ">=14" the newest available version 14 or higher

To cover several specific devices in one command, use a device matrix. Each cell is <device>[:<version>][:real], and every flow runs once per cell:

testingbot maestro sample.apk .maestro \
  --device-matrix "Pixel 9:15" \
  --device-matrix "Galaxy S24:14:real"

Variables, secrets and tags#

Variables use the same -e KEY=VALUE form as the Maestro CLI. Passwords and tokens should go through --secret-env instead: the value is stored encrypted and masked as *** in the API, reports, logs and dashboard. The same variable name cannot be passed as both.

testingbot maestro app.apk .maestro --device "Pixel 8" \
  -e API_URL=https://staging.example.com \
  --secret-env TEST_PASSWORD="$TEST_PASSWORD" \
  --include-tags smoke --exclude-tags flaky

Tag filtering behaves like Maestro's: with --include-tags, untagged flows are skipped; with --exclude-tags, they are kept. Filtered-out flows are never scheduled, so they do not use a session.

Retries, reports and artifacts#

testingbot maestro app.apk .maestro --device "Pixel 8" \
  --retry 1 \
  --report junit --report-output-dir ./reports \
  --download-artifacts failed --artifacts-output-dir ./artifacts

--retry re-runs failed flows up to 2 times and stops as soon as a flow passes. Reports can be junit, html, html-detailed or allure. Every run is also visible in the dashboard with video, logs and the steps of each flow.

Apps that talk to a private backend#

Cloud devices cannot reach localhost or a firewalled staging API by default. Add -t and the CLI starts a TestingBot Tunnel for the run, routes the device's traffic through your machine or CI runner, and closes the tunnel afterwards:

testingbot maestro app.apk .maestro --device "Pixel 8" -t

-t cannot be combined with --async, because the tunnel has to stay up for the whole run. The Maestro tunnel guide covers named tunnels for parallel builds.

Your app talks to a staging API behind the company firewall, and every cloud run fails at login. What do you add?

Cloud devices cannot reach private hosts on their own. The tunnel routes their traffic through your machine or runner; note that -t cannot be combined with --async.

Other run options#

Device locale (--device-locale de_DE), timezone (--timezone), orientation, network throttling (--throttle-network 3G) and IP geolocation (--geo-country-code DE) are flags too, which saves scripting them in flows. The Maestro options reference has the full list. If your flows currently run on Maestro Cloud, the migration guide maps each maestro cloud flag.

Next, wiring all of this into a CI pipeline.