---
title: Device Matrix, Parallel Runs and CI | TestingBot
description: One passing test on one phone proves very little. The introduction called
  fragmentation the defining problem of mobile app testing; this chapter turns that…
source_url:
  html: https://testingbot.com/resources/courses/mobile-app-testing/device-matrix-parallel-ci
  md: https://testingbot.com/resources/courses/mobile-app-testing/device-matrix-parallel-ci.md
---

Course

[Mobile App Testing: From First Test to CI](https://testingbot.com/resources/courses/mobile-app-testing)

Chapter 8 of 10

1. [1 Mobile App Testing: What Makes It Different](https://testingbot.com/resources/courses/mobile-app-testing)
2. [2 Emulators, Simulators and Real Devices](https://testingbot.com/resources/courses/mobile-app-testing/emulators-simulators-and-real-devices)
3. [3 Choosing a Framework: Appium, Espresso, XCUITest, Maestro](https://testingbot.com/resources/courses/mobile-app-testing/choosing-a-framework)
4. [4 Uploading and Managing App Builds](https://testingbot.com/resources/courses/mobile-app-testing/app-builds-and-uploads)
5. [5 Your First Appium Test on a Real Device](https://testingbot.com/resources/courses/mobile-app-testing/first-appium-test)
6. [6 Your First Maestro Flow on TestingBot](https://testingbot.com/resources/courses/mobile-app-testing/first-maestro-flow)
7. [7 Running Espresso and XCUITest Suites in the Cloud](https://testingbot.com/resources/courses/mobile-app-testing/espresso-and-xcuitest)
8. [8 Device Matrix, Parallel Runs and CI](https://testingbot.com/resources/courses/mobile-app-testing/device-matrix-parallel-ci)
9. [9 Debugging Failed Mobile Test Runs](https://testingbot.com/resources/courses/mobile-app-testing/debugging-failed-runs)
10. [10 Knowledge Check](https://testingbot.com/resources/courses/mobile-app-testing/knowledge-check)

[Exit course](https://testingbot.com/resources/courses)

 Chapter 8 of 10 Beginner 8 min read 2 quick checks Updated Nov 2026 

# Device Matrix, Parallel Runs and CI
On this page
1. [Build the matrix from your users](https://testingbot.com#build-the-matrix-from-your-users)
2. [One test, several configurations](https://testingbot.com#one-test-several-configurations)
3. [Exact names, patterns and what can go wrong](https://testingbot.com#exact-names-patterns-and-what-can-go-wrong)
4. [Parallel limits and queues](https://testingbot.com#parallel-limits-and-queues)
5. [Wiring it into CI](https://testingbot.com#wiring-it-into-ci)

One passing test on one phone proves very little. The introduction called fragmentation the defining problem of mobile app testing; this chapter turns that into a plan: which devices to test on, how to run them without waiting all day, and how to make it part of every build.

## Build the matrix from your users[#](https://testingbot.com#build-the-matrix-from-your-users)

Do not start from a list of every phone the cloud offers. Start from your analytics (Firebase, App Store Connect, Google Play Console, your own tracking) and write down:

1. **OS versions** that together cover most of your active users, plus your minimum supported version. The oldest supported version is where deprecated APIs and missing features bite.
2. **The top device models or families** , especially manufacturers with their own Android skin.
3. **Form factors** you support: small phone, large phone, tablet, foldable.
4. **Anything your app depends on** : a camera-heavy app needs real cameras; an app sold in many countries needs locales and right-to-left layouts.

Then split the list into tiers:

| Tier | When | Devices |
| --- | --- | --- |
| Smoke | Every pull request | One Android emulator, one iOS simulator, newest OS |
| Regression | Nightly or on merge | Your top 4-8 combinations, at least half on real devices |
| Release | Before shipping | The full matrix on real devices, including the oldest supported OS |

This keeps pull request feedback fast and still puts every release on the hardware your users have.

Your analytics show 40% of Android users on Samsung phones and your minimum version is Android 10. The nightly matrix has three Pixels on Android 15 and 16. What is the most important change?

 Add more Pixels on Android 16 Replace the Pixels with emulators to save time Add real Samsung devices and a device on the oldest supported Android version Drop Android 10 support

The matrix should mirror your users. It currently misses the largest manufacturer, with its own Android skin, and the oldest supported version, where compatibility bugs collect. More of the same Pixel adds little.

## One test, several configurations[#](https://testingbot.com#one-test-several-configurations)

The script below runs the calculator check from chapter 4 on three configurations: an Android emulator, a real Android phone and an iOS simulator. The only per-platform difference is how a field is located; the capabilities come from a list.

```javascript
// matrix.mjs: the same check on several device configurations
import { remote } from 'webdriverio'

const ANDROID_APP = 'https://testingbot.com/appium/sample.apk'
const IOS_SIM_APP = 'https://testingbot.com/appium/sample.zip'

const matrix = [
  { platformName: 'Android', 'appium:deviceName': 'Pixel 8',
    'appium:platformVersion': '14', 'appium:app': ANDROID_APP },
  { platformName: 'Android', 'appium:deviceName': 'Pixel [89]',
    'appium:app': ANDROID_APP, real: true },
  { platformName: 'iOS', 'appium:deviceName': 'iPhone 16',
    'appium:platformVersion': '18.6', 'appium:app': IOS_SIM_APP }
]

// iOS exposes the field ids on more than one element, so pin the type.
const field = (driver, id) => driver.isAndroid
  ? driver.$(`~${id}`)
  : driver.$(`-ios predicate string:name == '${id}' AND type == 'XCUIElementTypeTextField'`)

async function run ({ real = false, ...caps }) {
  const driver = await remote({
    hostname: 'hub.testingbot.com', protocol: 'https', port: 443, path: '/wd/hub',
    user: process.env.TB_KEY, key: process.env.TB_SECRET, logLevel: 'warn',
    connectionRetryTimeout: 420000, connectionRetryCount: 0,
    capabilities: {
      ...caps,
      'appium:automationName': caps.platformName === 'iOS' ? 'XCUITest' : 'UiAutomator2',
      'tb:options': { realDevice: real, name: 'Calculator matrix', build: process.env.BUILD ?? 'matrix' }
    }
  })
  const sessionId = driver.sessionId // deleteSession() clears it
  let passed = false
  try {
    await field(driver, 'inputA').setValue('10')
    await field(driver, 'inputB').setValue('5')
    passed = (await field(driver, 'sum').getText()) === '15'
  } finally {
    await driver.execute(`tb:test-result=${passed ? 'passed' : 'failed'}`)
    await driver.deleteSession()
  }
  if (!passed) process.exitCode = 1
  return `${caps['appium:deviceName']} (${real ? 'real' : 'virtual'}): ${passed ? 'passed' : 'failed'} ${sessionId}`
}

// Never start more sessions than your plan runs in parallel.
const PARALLEL = Number(process.env.PARALLEL ?? 2)
for (let i = 0; i < matrix.length; i += PARALLEL) {
  const batch = matrix.slice(i, i + PARALLEL).map(run)
  // allSettled: one device that fails to start must not hide the others' results.
  for (const r of await Promise.allSettled(batch)) {
    if (r.status === 'rejected') process.exitCode = 1
    console.log(r.status === 'fulfilled' ? r.value : `error: ${r.reason.message.slice(0, 120)}`)
  }
}
```

```bash
PARALLEL=1 BUILD=mobile-app-testing-course node matrix.mjs
```

```
Pixel 8 (virtual): passed a3df5e1b0d00-017313ed099c-f062ac728627-179450223165-4439369
Pixel [89] (real): passed 1d0ffc4cc41a-2f2e01afed82-dcb14b6b5ee1-179450224772-51425966
iPhone 16 (virtual): passed 170567972acb-810299f8b890-2d23d67f1c54-179450228990-03614852
```

A failed check or a device that cannot be started sets `process.exitCode = 1`, so the job fails in CI. With one row expecting the wrong sum and one row asking for a device that does not exist, the script printed `Pixel 8 (virtual): failed c0a3cab9a09d-...` and the grid's "not available" error, and exited with 1.

The course ran it one session at a time (`PARALLEL=1`) because the account was shared with other work. In your own pipeline, set `PARALLEL` to your plan's limit. A test runner such as WebdriverIO does the same job with `capabilities` and `maxInstances`, as the [Appium course](https://testingbot.com/resources/courses/appium-webdriverio/tests-on-testingbot) shows; the script just makes the mechanics visible.

## Exact names, patterns and what can go wrong[#](https://testingbot.com#exact-names-patterns-and-what-can-go-wrong)

A device can be requested two ways, and the matrix above uses both:

- **Exact name and version** (`Pixel 8` + `14`, `iPhone 16` + `18.6`): reproducible. Use it when a bug report names a device, and for every row that represents a real user segment.
- **A pattern** (`Pixel [89]`): the first free match, so less waiting for a busy phone. Use it when "any recent Pixel" is good enough.

Keep patterns narrow. While writing this course, an earlier version of the script asked for `Pixel.*` without a version. That pattern also matches the original Pixel, and one session landed on it, running Android 7.1, and failed while installing the app. A broad pattern trades waiting time for an unpredictable OS version; constrain the name, add a version, or both.

Your nightly job asks for `Galaxy.*` on real devices. Results flip between green and red for no visible reason. What should you check first?

 Which device and OS version each run actually got; the pattern may be matching very different phones Whether the app download is too slow Whether `PARALLEL` is set too high Whether the tests need longer waits

A broad pattern can land on a tablet one night and a five-year-old phone the next. Compare the device and version of passing and failing runs before touching waits; then narrow the pattern or pin a version.

## Parallel limits and queues[#](https://testingbot.com#parallel-limits-and-queues)

Your plan allows a fixed number of sessions at the same time. Requests beyond that are queued and start as slots free up; they are not rejected. Real devices add a second kind of wait: a specific model may be in use, and every device is cleaned after each session before the next one can start.

Practical consequences:

- Total suite time is roughly _tests x average test length / parallel slots_. More slots, shorter tests or sharding are the only ways to cut it. [Parallel testing](https://testingbot.com/features/parallel-testing) explains how to size this.
- Tests must not depend on each other or on order: in parallel, any test may run first, on any device.
- Give session creation a long timeout (the `connectionRetryTimeout` above), so queued sessions are not counted as failures.

The TestingBot CLI does this for Maestro, Espresso and XCUITest runs: Maestro runs each flow in its own session by default, up to your limit, and `--device-matrix` and `--shard-split` control how flows spread over devices (see the [Maestro course chapter on TestingBot runs](https://testingbot.com/resources/courses/maestro/run-on-testingbot)). `--priority high` lets a release candidate overtake your other waiting runs; it does not raise the limit.

## Wiring it into CI[#](https://testingbot.com#wiring-it-into-ci)

A mobile CI job has the same four steps whatever the framework:

1. **Build** the app (and the test APK or test bundle for native suites).
2. **Upload** the fresh build: to TestingBot Storage for Appium (chapter 3), or let the CLI do it for Maestro, Espresso and XCUITest.
3. **Run** the tier that fits the trigger, with the commit or build number in the build name.
4. **Publish results** : JUnit reports (`--report junit` with the CLI, a JUnit reporter in your test runner), and fail the job on test failures.

The CLI's exit codes make step 4 dependable. This course ran the failing Espresso test from chapter 6 on its own:

```bash
npx @testingbot/cli@1.5.0 espresso app-debug.apk app-debug-androidTest.apk \
  --device "Pixel 8" --platform-version "14" \
  --class com.testingbot.calculator.AdvancedCalculatorTest \
  --build "course-mobile-app-testing" \
  --download-artifacts failed --artifacts-output-dir ./artifacts
echo "exit=$?"
```

```
Tests run: 1, Failures: 1
  Run 3258 (Pixel 8): Test failed
1 test run(s) failed:
Downloading artifacts for 1 run(s)...
Artifacts saved to: artifacts/espresso_artifacts_2026-11-12T15-38-03-534Z.zip
exit=2
```

Exit code 2 means tests ran and failed, exit code 1 means the CLI or the infrastructure had a problem, and 0 means everything passed. A pipeline can retry on 1 and report on 2. Inside GitHub Actions and Azure Pipelines the CLI also writes a results table into the job summary. For ready-made workflow files, see the [Maestro CI chapter](https://testingbot.com/resources/courses/maestro/ci-pipelines) and TestingBot's [Maestro CI/CD docs](https://testingbot.com/support/app-automate/maestro/ci-cd).

[Previous · Chapter 7 Running Espresso and XCUITest Suites in the Cloud](https://testingbot.com/resources/courses/mobile-app-testing/espresso-and-xcuitest) [Next · Chapter 9 Debugging Failed Mobile Test Runs](https://testingbot.com/resources/courses/mobile-app-testing/debugging-failed-runs)

On this page

1. [Build the matrix from your users](https://testingbot.com#build-the-matrix-from-your-users)
2. [One test, several configurations](https://testingbot.com#one-test-several-configurations)
3. [Exact names, patterns and what can go wrong](https://testingbot.com#exact-names-patterns-and-what-can-go-wrong)
4. [Parallel limits and queues](https://testingbot.com#parallel-limits-and-queues)
5. [Wiring it into CI](https://testingbot.com#wiring-it-into-ci)

Certification

Earn the free, resume-worthy TestingBot Mobile App Testing Certification for this course.

[How it works](https://testingbot.com/resources/courses/mobile-app-testing/certification)
