Device Matrix, Parallel Runs and CI
On this page
One passing test on one phone proves very little. The introduction called fragmentation the defining problem of mobile app testing; this chapter turns that into a plan: which devices to test on, how to run them without waiting all day, and how to make it part of every build.
Build the matrix from your users#
Do not start from a list of every phone the cloud offers. Start from your analytics (Firebase, App Store Connect, Google Play Console, your own tracking) and write down:
- OS versions that together cover most of your active users, plus your minimum supported version. The oldest supported version is where deprecated APIs and missing features bite.
- The top device models or families, especially manufacturers with their own Android skin.
- Form factors you support: small phone, large phone, tablet, foldable.
- Anything your app depends on: a camera-heavy app needs real cameras; an app sold in many countries needs locales and right-to-left layouts.
Then split the list into tiers:
| Tier | When | Devices |
|---|---|---|
| Smoke | Every pull request | One Android emulator, one iOS simulator, newest OS |
| Regression | Nightly or on merge | Your top 4-8 combinations, at least half on real devices |
| Release | Before shipping | The full matrix on real devices, including the oldest supported OS |
This keeps pull request feedback fast and still puts every release on the hardware your users have.
Your analytics show 40% of Android users on Samsung phones and your minimum version is Android 10. The nightly matrix has three Pixels on Android 15 and 16. What is the most important change?
The matrix should mirror your users. It currently misses the largest manufacturer, with its own Android skin, and the oldest supported version, where compatibility bugs collect. More of the same Pixel adds little.
One test, several configurations#
The script below runs the calculator check from chapter 4 on three configurations: an Android emulator, a real Android phone and an iOS simulator. The only per-platform difference is how a field is located; the capabilities come from a list.
// matrix.mjs: the same check on several device configurations
import { remote } from 'webdriverio'
const ANDROID_APP = 'https://testingbot.com/appium/sample.apk'
const IOS_SIM_APP = 'https://testingbot.com/appium/sample.zip'
const matrix = [
{ platformName: 'Android', 'appium:deviceName': 'Pixel 8',
'appium:platformVersion': '14', 'appium:app': ANDROID_APP },
{ platformName: 'Android', 'appium:deviceName': 'Pixel [89]',
'appium:app': ANDROID_APP, real: true },
{ platformName: 'iOS', 'appium:deviceName': 'iPhone 16',
'appium:platformVersion': '18.6', 'appium:app': IOS_SIM_APP }
]
// iOS exposes the field ids on more than one element, so pin the type.
const field = (driver, id) => driver.isAndroid
? driver.$(`~${id}`)
: driver.$(`-ios predicate string:name == '${id}' AND type == 'XCUIElementTypeTextField'`)
async function run ({ real = false, ...caps }) {
const driver = await remote({
hostname: 'hub.testingbot.com', protocol: 'https', port: 443, path: '/wd/hub',
user: process.env.TB_KEY, key: process.env.TB_SECRET, logLevel: 'warn',
connectionRetryTimeout: 420000, connectionRetryCount: 0,
capabilities: {
...caps,
'appium:automationName': caps.platformName === 'iOS' ? 'XCUITest' : 'UiAutomator2',
'tb:options': { realDevice: real, name: 'Calculator matrix', build: process.env.BUILD ?? 'matrix' }
}
})
const sessionId = driver.sessionId // deleteSession() clears it
let passed = false
try {
await field(driver, 'inputA').setValue('10')
await field(driver, 'inputB').setValue('5')
passed = (await field(driver, 'sum').getText()) === '15'
} finally {
await driver.execute(`tb:test-result=${passed ? 'passed' : 'failed'}`)
await driver.deleteSession()
}
if (!passed) process.exitCode = 1
return `${caps['appium:deviceName']} (${real ? 'real' : 'virtual'}): ${passed ? 'passed' : 'failed'} ${sessionId}`
}
// Never start more sessions than your plan runs in parallel.
const PARALLEL = Number(process.env.PARALLEL ?? 2)
for (let i = 0; i < matrix.length; i += PARALLEL) {
const batch = matrix.slice(i, i + PARALLEL).map(run)
// allSettled: one device that fails to start must not hide the others' results.
for (const r of await Promise.allSettled(batch)) {
if (r.status === 'rejected') process.exitCode = 1
console.log(r.status === 'fulfilled' ? r.value : `error: ${r.reason.message.slice(0, 120)}`)
}
}
PARALLEL=1 BUILD=mobile-app-testing-course node matrix.mjs
Pixel 8 (virtual): passed a3df5e1b0d00-017313ed099c-f062ac728627-179450223165-4439369
Pixel [89] (real): passed 1d0ffc4cc41a-2f2e01afed82-dcb14b6b5ee1-179450224772-51425966
iPhone 16 (virtual): passed 170567972acb-810299f8b890-2d23d67f1c54-179450228990-03614852
A failed check or a device that cannot be started sets process.exitCode = 1, so the job fails in CI. With one row expecting the wrong sum and one row asking for a device that does not exist, the script printed Pixel 8 (virtual): failed c0a3cab9a09d-... and the grid's "not available" error, and exited with 1.
The course ran it one session at a time (PARALLEL=1) because the account was shared with other work. In your own pipeline, set PARALLEL to your plan's limit. A test runner such as WebdriverIO does the same job with capabilities and maxInstances, as the Appium course shows; the script just makes the mechanics visible.
Exact names, patterns and what can go wrong#
A device can be requested two ways, and the matrix above uses both:
-
Exact name and version (
Pixel 8+14,iPhone 16+18.6): reproducible. Use it when a bug report names a device, and for every row that represents a real user segment. -
A pattern (
Pixel [89]): the first free match, so less waiting for a busy phone. Use it when "any recent Pixel" is good enough.
Keep patterns narrow. While writing this course, an earlier version of the script asked for Pixel.* without a version. That pattern also matches the original Pixel, and one session landed on it, running Android 7.1, and failed while installing the app. A broad pattern trades waiting time for an unpredictable OS version; constrain the name, add a version, or both.
Your nightly job asks for Galaxy.* on real devices. Results flip between green and red for no visible reason. What should you check first?
A broad pattern can land on a tablet one night and a five-year-old phone the next. Compare the device and version of passing and failing runs before touching waits; then narrow the pattern or pin a version.
Parallel limits and queues#
Your plan allows a fixed number of sessions at the same time. Requests beyond that are queued and start as slots free up; they are not rejected. Real devices add a second kind of wait: a specific model may be in use, and every device is cleaned after each session before the next one can start.
Practical consequences:
- Total suite time is roughly tests x average test length / parallel slots. More slots, shorter tests or sharding are the only ways to cut it. Parallel testing explains how to size this.
- Tests must not depend on each other or on order: in parallel, any test may run first, on any device.
- Give session creation a long timeout (the
connectionRetryTimeoutabove), so queued sessions are not counted as failures.
The TestingBot CLI does this for Maestro, Espresso and XCUITest runs: Maestro runs each flow in its own session by default, up to your limit, and --device-matrix and --shard-split control how flows spread over devices (see the Maestro course chapter on TestingBot runs). --priority high lets a release candidate overtake your other waiting runs; it does not raise the limit.
Wiring it into CI#
A mobile CI job has the same four steps whatever the framework:
- Build the app (and the test APK or test bundle for native suites).
- Upload the fresh build: to TestingBot Storage for Appium (chapter 3), or let the CLI do it for Maestro, Espresso and XCUITest.
- Run the tier that fits the trigger, with the commit or build number in the build name.
-
Publish results: JUnit reports (
--report junitwith the CLI, a JUnit reporter in your test runner), and fail the job on test failures.
The CLI's exit codes make step 4 dependable. This course ran the failing Espresso test from chapter 6 on its own:
npx @testingbot/cli@1.5.0 espresso app-debug.apk app-debug-androidTest.apk \
--device "Pixel 8" --platform-version "14" \
--class com.testingbot.calculator.AdvancedCalculatorTest \
--build "course-mobile-app-testing" \
--download-artifacts failed --artifacts-output-dir ./artifacts
echo "exit=$?"
Tests run: 1, Failures: 1
Run 3258 (Pixel 8): Test failed
1 test run(s) failed:
Downloading artifacts for 1 run(s)...
Artifacts saved to: artifacts/espresso_artifacts_2026-11-12T15-38-03-534Z.zip
exit=2
Exit code 2 means tests ran and failed, exit code 1 means the CLI or the infrastructure had a problem, and 0 means everything passed. A pipeline can retry on 1 and report on 2. Inside GitHub Actions and Azure Pipelines the CLI also writes a results table into the job summary. For ready-made workflow files, see the Maestro CI chapter and TestingBot's Maestro CI/CD docs.