Exit course
Chapter 8 of 10 Intermediate 8 min read 2 quick checks Updated

Run Espresso Tests on TestingBot

On this page
  1. How a cloud Espresso run works
  2. Run with the CLI
  3. Run part of the suite
  4. Run with the REST API
  5. Read the results

An emulator on your laptop is one device, one Android version and one screen size. Your users have hundreds. This chapter takes the two APKs from chapter 6 and runs them on TestingBot's Android emulators and real devices: first with the CLI, then with the REST API, including filters, sharding and the build name that groups runs in the dashboard.

How a cloud Espresso run works#

There is no WebDriver session and no test code running on your machine. You hand over the two files and say where to run them:

  1. Upload the app (.apk or .aab) and the test APK.
  2. Start a run with one or more device capabilities. Each capability becomes its own run on its own device.
  3. TestingBot installs both APKs on the device and starts the instrumentation, by default ${packageName}/androidx.test.runner.AndroidJUnitRunner, the same thing you did with adb shell am instrument in chapter 6.
  4. Read the results: pass or fail per run, the instrumentation output, a video, logs and a JUnit XML report.

Because step 3 is a plain instrumentation run, the lesson from chapter 5 applies: Gradle's Orchestrator settings do not travel with the APKs, so the suite must reset its own state. Espresso Shop's does.

Run with the CLI#

The TestingBot CLI does the upload, the start and the waiting in one command. Install it with npm install -g @testingbot/cli and authenticate with testingbot login, or, as this course does, run a pinned version through npx with the key and secret in ~/.testingbot or in the TB_KEY and TB_SECRET environment variables.

List the devices that accept Espresso runs:

npx @testingbot/cli@1.5.0 devices --framework espresso
PLATFORM  DEVICE             TYPE       VERSIONS
Android   Galaxy S24         phone      14.0
Android   Pixel 8            phone      17.0, 14.0
Android   Pixel 9            phone      16.0, 15.0
...

Then run the suite on a Pixel 9 emulator with Android 15:

npx @testingbot/cli@1.5.0 espresso \
  app/build/outputs/apk/debug/app-debug.apk \
  app/build/outputs/apk/androidTest/debug/app-debug-androidTest.apk \
  --device "Pixel 9" --platform-version "15" \
  --name "Espresso course" --build "espresso-course" \
  --report junit --report-output-dir ./reports

The CLI uploads both files, starts the run and streams the instrumentation output while it waits. The end of this course's run:

com.example.shop.HelpLinkTest:
com.example.shop.LoginScreenTest:
com.example.shop.LoginTest:
com.example.shop.ProductListTest:
com.example.shop.RobotLoginTest:

Time: 14.417

OK (10 tests)
  Run 3264 (Pixel 9): Test completed successfully
All tests completed successfully!
Saved report: reports/espresso_report_3243.xml

All ten tests passed on the Pixel 9 emulator with Android 15 (session 728addd8979c-...), including the login tests that depend on the IdlingResource and the list tests that depend on EmptyBasketRule. The CLI exits with 0 when every test passed and with a non-zero code otherwise, which is what CI needs. The saved file is the JUnit XML report.

The options that matter most:

Option What it does
--device, --platform-version Which device and Android version. --device accepts a regular expression, such as "Pixel.*".
--real-device A physical device instead of an emulator, for example --device "Galaxy S24" --real-device
--phone-only, --tablet-only Restrict a wildcard device name to phones or tablets
--name The name of the run in the dashboard
--build The build the run belongs to, which groups runs in the dashboard
--report junit Download the JUnit XML report into --report-output-dir
--download-artifacts Download video, logs and screenshots after the run (failed for failed runs only)
--test-runner A custom instrumentation runner, if testInstrumentationRunner is not AndroidJUnitRunner

The build name#

--build deserves a second look. The CLI sends it as the build capability of the run, and TestingBot files every session of that run under that build. Leave it out and the run lands in a default build called "Espresso Project" plus a number, one per upload, which quickly becomes a long list of builds nobody can tell apart. Use something that identifies what was tested: a version (--build "v2.1.0"), a CI run (--build "espresso-shop-1234"), or, for this course's certificate, --build certificate-espresso. --name is different: it labels the run, not the build.

You ran the suite on three devices from three separate CLI commands and want all three runs together in the dashboard. What do you pass?

The build groups runs. Each upload gets its own default build, so three commands without --build produce three builds. --name only labels the individual run.

Run part of the suite#

The instrumentation filters from chapter 6 are CLI options too: --class, --not-class, --package, --not-package, --annotation, --not-annotation and --size, with comma-separated values. Running only the two login classes on a Pixel 8 with Android 14:

npx @testingbot/cli@1.5.0 espresso \
  app/build/outputs/apk/debug/app-debug.apk \
  app/build/outputs/apk/androidTest/debug/app-debug-androidTest.apk \
  --device "Pixel 8" --platform-version "14" \
  --class "com.example.shop.LoginTest,com.example.shop.RobotLoginTest" \
  --name "Espresso course, login tests" --build "espresso-course"
com.example.shop.LoginTest:
com.example.shop.RobotLoginTest:

Time: 14.523

OK (4 tests)
  Run 3273 (Pixel 8): Test completed successfully
All tests completed successfully!

Four tests instead of ten (session ad9fd6273332-...). The same espresso-course build now holds runs on two devices, which is what the build name is for.

Run with the REST API#

The CLI is a wrapper around a small REST API, which is what you use from a CI system without Node.js or from your own tooling. The Espresso documentation has every call; the flow is three requests. Authentication is HTTP Basic with your key and secret; the commands below read them from TB_KEY and TB_SECRET.

Upload the app. The response contains the id of a new Espresso project:

curl -u "$TB_KEY:$TB_SECRET" \
  -X POST "https://api.testingbot.com/v1/app-automate/espresso/app" \
  -F "file=@app/build/outputs/apk/debug/app-debug.apk"
{"id":3246}

Upload the test APK to that project:

curl -u "$TB_KEY:$TB_SECRET" \
  -X POST "https://api.testingbot.com/v1/app-automate/espresso/3246/tests" \
  -F "file=@app/build/outputs/apk/androidTest/debug/app-debug-androidTest.apk"

Start the run. Every entry in capabilities is one device; espressoOptions carries the filters (here: only the product list tests); and the build name goes inside each capability:

curl -u "$TB_KEY:$TB_SECRET" \
  -X POST "https://api.testingbot.com/v1/app-automate/espresso/3246/run" \
  -H "Content-Type: application/json" \
  -d '{
    "capabilities": [
      { "platformName": "Android", "deviceName": "Pixel 9", "version": "15",
        "build": "espresso-course" }
    ],
    "espressoOptions": { "class": ["com.example.shop.ProductListTest"] }
  }'
{"success":true,"id":3246,"runs":[{"id":3276,"capabilities":{"platformName":"Android","deviceName":"Pixel 9","version":"15","build":"espresso-course"}}],"priority":"normal"}

Each capability became a run with its own id (3276).

Poll the project for its status, and fetch the JUnit report once it is done:

curl -u "$TB_KEY:$TB_SECRET" "https://api.testingbot.com/v1/app-automate/espresso/3246"
curl -u "$TB_KEY:$TB_SECRET" "https://api.testingbot.com/v1/app-automate/espresso/3246/report"

The status response, shortened:

{"runs":[{"id":3276,"status":"DONE","success":true,
  "capabilities":{"platformName":"Android","deviceName":"Pixel 9","version":"15","build":"espresso-course"},
  "report":"<testsuite name=\"EspressoRun #3276\" tests=\"4\" failures=\"0\" errors=\"0\" time=\"4.174\"> ...",
  "test":{"sessionId":"ca932e6b90fb-...","environment":{"os":"Pixel 9 - 15.0","version":"15.0"}}}],
 "success":true,"completed":true}

tests="4" confirms the filter: only the four ProductListTest tests ran. success is true only when every test in the run passed, and sessionId links the run to its session in the dashboard.

Splitting a suite#

The filter documentation describes numShards and shardIndex for splitting a suite, one run per shard index; the CLI has no sharding option. Whatever you use to split a suite, check the tests count in each part's report, as in the response above: the counts over all parts must add up to the suite size, or tests are being run twice or not at all. Splitting by class or package with the filters is the most predictable way to divide a suite across devices, because you decide which tests go where. Remember from chapter 6 that the runner's own sharding assigns tests by a hash of their names, so it pays off for hundreds of tests, not for ten.

Read the results#

Every run appears in the TestingBot dashboard as a session with:

  • the video of the device during the run, which is often the quickest way to see why a check failed (a dialog in the way, a keyboard covering the button),
  • the instrumentation output, with the same Espresso error messages and view hierarchy dumps you get locally,
  • device logs (logcat), for crashes and anything the app logs,
  • the build and run name you set.

A run passes when every test in it passed. For one failed test, re-run that class locally first with adb shell am instrument -e class ... on an emulator of the same Android version; most failures reproduce there. If it only fails in the cloud, look at what differs: the device's screen size (a view pushed off screen), the Android version (a permission or behaviour change), or state the suite relied on. The Espresso test reports page describes the report formats, and the Mobile App Testing course has a chapter on reading failed cloud runs.

A test passes on your emulator and fails on a small real phone in the cloud with "not displayed enough to be clicked". What do you look at first?

The error says the view was not visible enough, which on a smaller screen usually means the keyboard or the layout hides it. The video shows it directly; the fix is often a closeSoftKeyboard() or a scroll.