Exit course
Chapter 9 of 10 Beginner 6 min read 2 quick checks Updated

Debugging Failed Mobile Test Runs

On this page
  1. Where the evidence is
  2. A failure, step by step
  3. Getting the logs
  4. Mobile-specific failures and their usual causes
  5. Make the next failure easier

A failed run on a device you cannot touch is only debuggable if you know where the evidence is. Every TestingBot session keeps a video of the screen and several logs; the skill is reading them in the right order. This chapter works through a real failure from chapter 6 and then covers the failures that are specific to mobile.

Where the evidence is#

Source Answers
Your test output and JUnit report Which test failed and which assertion
Video and screenshots What the screen showed at that moment
Framework log (Appium, Espresso instrumentation, Maestro) Which command failed and what it was told
Device log (Android logcat, iOS device log) What the app and OS did: crashes, exceptions, permission denials

Start at the top and go down only as far as you need. Most failures are explained by the assertion message plus ten seconds of video; the device log is for crashes and "nothing happened" cases.

A failure, step by step#

Chapter 6 ran the Espresso demo suite and one test failed. Run on its own (the CI example in chapter 7), the CLI printed the assertion:

Expected: with text: is "50e.0"
     Got: "MaterialTextView{id=2131231009, res-name=output, visibility=VISIBLE, ... text=500.0, ...}"
FAILURES!!!
Tests run: 1,  Failures: 1

Reading it:

  1. Which element? res-name=output: the calculator's result field, found and visible. Locators are fine.
  2. What did it show? text=500.0. The test, ensureLargeNumbersWork, typed 1000000, divided by 2000 and pressed equals: 500.0 is the right answer.
  3. What did the test expect? 50e.0, which is not a number any calculator shows. The bug is a typo in the test's expected value, not in the app.

No video or device log was needed: the assertion message contained both values. That is the payoff of assertions that report expected and actual values; a bare assert(passed) would have sent you to the video.

An assertion fails with "expected the order list to contain 3 rows, found 0". The video shows a spinner on that screen when the test fails. What is the most likely story?

A spinner means the screen was still loading, so zero rows was true at that instant. Wait for a loaded state, such as the first row or the spinner disappearing. A wrong locator would also find nothing after loading, and a crash would show the home screen or a crash dialog, not a spinner.

Getting the logs#

After a CLI run, --download-artifacts failed (shown in chapter 7) saves a zip per failed run. For the failed Espresso run it contained:

3258_Pixel_8/logs/android.txt
3258_Pixel_8/logs/appium.txt
3258_Pixel_8/logs/espresso.txt
3258_Pixel_8/logs/logcat.txt
3258_Pixel_8/logs/vm.txt
3258_Pixel_8/video/video.mp4

For a Maestro run that already finished, the CLI fetches the same kind of bundle by project id, including a maestro.txt log per flow:

npx @testingbot/cli@1.5.0 artifacts --id 142983 \
  --download-artifacts --artifacts-output-dir ./artifacts

For Appium sessions, or to script it yourself, the REST API returns the session with links to its video and logs. The log files are gzip-compressed:

SESSION=894e8d77c9a3-0b84c342dea6-5db392060332-179449712832-68310315
curl -s -u "$TB_KEY:$TB_SECRET" "https://api.testingbot.com/v1/tests/$SESSION" \
  | jq -c '{success, device_name, platform_name, logs: (.logs | keys)}'
{"success":false,"device_name":"Pixel 8","platform_name":"Android","logs":["android","appium","espresso","logcat","vm"]}
curl -s -u "$TB_KEY:$TB_SECRET" "https://api.testingbot.com/v1/tests/$SESSION" \
  | jq -r '.logs.logcat' \
  | xargs curl -s | gunzip > logcat.txt
grep -m 3 "TestRunner" logcat.txt
11-12 15:25:28.747  5594  5649 I TestRunner: run started: 6 tests
11-12 15:25:28.748  5594  5649 I TestRunner: started: ensureLargeNumbersWork(com.testingbot.calculator.AdvancedCalculatorTest)
11-12 15:25:33.481  5594  5649 E TestRunner: failed: ensureLargeNumbersWork(com.testingbot.calculator.AdvancedCalculatorTest)

The links in the API response are signed and expire after a while, so download what you need when the run finishes rather than storing the links. In the dashboard, the same session page plays the video next to the list of commands, which is usually the fastest view for a person.

An Android test fails because the app suddenly shows the home screen in the middle of a flow. The Appium log shows the commands succeeding until the element is missing. Where do you look next?

Jumping to the home screen usually means the app process died. The framework only sees that elements disappeared; the crash stack trace or the OS killing the process is in logcat. Waits cannot fix a crash.

Mobile-specific failures and their usual causes#

  • Session never starts, or fails installing the app. Check the artefact (chapter 3: right file for the target, CPU architecture, not expired), the device you actually got (a broad pattern can land on an old OS, as chapter 7 showed) and, for public URLs, whether the device cloud can download the file.
  • Fails on real devices, passes on emulators. Look for vendor dialogs, permission prompts, a different keyboard covering the button, or slower start-up on a mid-range phone. The video shows most of these immediately.
  • A system dialog covers the screen. Permissions, "app isn't responding", battery or update prompts. Grant permissions up front where the test is not about them; see Handling permission pop-ups.
  • Passes alone, fails in the suite. Leaked state from an earlier test: logged-in user, data on the device, an open dialog. Start each test from a clean app state (clearState in Maestro, a fresh session or app reset in Appium).
  • Flaky only on some runs. Compare the device, OS version and duration of passing and failing runs first. Timing problems show up as failures on the slowest devices; device problems as failures on one model.
  • The app cannot reach its backend. Cloud devices cannot see localhost or a firewalled staging API. Use TestingBot Tunnel: -t with the CLI, or the tunnel option in your test runner; the Local Testing course explains it.

Make the next failure easier#

Every failure you debug is a chance to make the next one cheaper: assertions that print expected and actual values, a build name with the commit, test names that say what they check, and screenshots at the steps that matter. The Espresso demo's takeActivityScreenshot("addition_works") calls are an example; TestingBot's Espresso screenshot guide shows how they end up on the session page.