Exit course
Chapter 7 of 10 Intermediate 6 min read 2 quick checks Updated

Building the App APK and the Test APK

On this page
  1. Two APKs from one project
  2. Run them without Gradle
  3. Running part of the suite
  4. Sharding

On your machine, ./gradlew connectedDebugAndroidTest hides a lot: it builds two APKs, installs them, starts the instrumentation and collects the results. A device cloud, a CI system or a colleague's phone gets none of that for free. They get the two files, and the instrumentation does the rest. This chapter builds those files, looks inside them, and runs them the way any other system will: with adb and no Gradle.

Two APKs from one project#

./gradlew assembleDebug assembleDebugAndroidTest
Task Output Contains
assembleDebug app/build/outputs/apk/debug/app-debug.apk Your app, exactly as a user would install it in debug mode
assembleDebugAndroidTest app/build/outputs/apk/androidTest/debug/app-debug-androidTest.apk Your test classes, Espresso and the other test libraries, and the instrumentation declaration

For Espresso Shop both came out at about 3.5 MB. The test APK has its own package name, the app's id plus .test, and its manifest declares which app it instruments. Dumped with aapt2 from the Android build tools:

$ aapt2 dump xmltree --file AndroidManifest.xml app-debug-androidTest.apk
  E: instrumentation (line=9)
    A: android:label="Tests for com.example.shop"
    A: android:name="androidx.test.runner.AndroidJUnitRunner"
    A: android:targetPackage="com.example.shop"

That <instrumentation> element is how Android knows that installing com.example.shop.test gives it a runner, AndroidJUnitRunner, for the app com.example.shop. It is generated from testInstrumentationRunner in build.gradle.kts.

A few rules that save debugging time:

  • Both APKs must be signed with the same key. Gradle signs both debug builds with your debug keystore, so this is automatic for debug. If you build the app in CI with another key, build the test APK in the same job.
  • They must come from the same build. The test APK is compiled against the app's classes and R.id values. A test APK from yesterday's commit with today's app can fail in confusing ways, such as views that are not found although the screen clearly has them.
  • Test the debug build type unless you have a reason not to. Instrumented tests run against debug by default. To test another build type, set testBuildType = "staging" in the android block; Gradle then generates assembleStagingAndroidTest. Minified builds need extra ProGuard/R8 rules to keep the classes your tests call, which is why most teams test an unminified variant.

CI builds the release-signed app in one job and the test APK in another job with the debug key. Installing works, but the run fails before any test starts. What is the likely cause?

Instrumentation requires the test APK and the target app to share a signing certificate. Build both in the same job with the same signing configuration. Espresso can test any build type, as long as it is built for testing.

Run them without Gradle#

Install both APKs on a running emulator or device and list the instrumentations Android now knows about:

adb install -r -t app/build/outputs/apk/debug/app-debug.apk
adb install -r -t app/build/outputs/apk/androidTest/debug/app-debug-androidTest.apk
adb shell pm list instrumentation
instrumentation:com.example.shop.test/androidx.test.runner.AndroidJUnitRunner (target=com.example.shop)

(-r replaces an installed version, -t allows test-only packages.) Run the whole suite:

adb shell am instrument -w com.example.shop.test/androidx.test.runner.AndroidJUnitRunner
Time: 28.811

OK (10 tests)

This is essentially what a device cloud does on its device: install two APKs and start the runner. If a suite passes this way on an emulator, it is ready to leave your machine. If it only passes through Gradle, find out why first; the usual answer is state that only Orchestrator was clearing (chapter 5).

Running part of the suite#

AndroidJUnitRunner takes filter arguments with -e. The ones you will use:

Argument Runs
-e class com.example.shop.LoginTest One class; add #methodName for one method; comma-separate several
-e notClass com.example.shop.HelpLinkTest Everything except that class
-e package com.example.shop.checkout Every test in a package
-e size medium Tests annotated @SmallTest, @MediumTest or @LargeTest
-e annotation com.example.shop.Smoke Tests with your own annotation
-e numShards 2 -e shardIndex 0 One shard of the suite

Size annotations come from androidx.test.filters and go on the class or the method. In Espresso Shop the list and intent tests are @MediumTest and the login tests are @LargeTest:

@MediumTest
@RunWith(AndroidJUnit4::class)
class ProductListTest { ... }

Running only the medium tests:

adb shell am instrument -w -e size medium \
  com.example.shop.test/androidx.test.runner.AndroidJUnitRunner
com.example.shop.HelpLinkTest:.
com.example.shop.ProductListTest:....
OK (5 tests)

The same filters work through Gradle as properties, for example ./gradlew connectedDebugAndroidTest -Pandroid.testInstrumentationRunnerArguments.size=medium, and on TestingBot as options, as the next chapter shows.

Sharding#

Sharding splits a suite into parts that run in parallel on different devices. Each device runs the same APKs with the same numShards and a different shardIndex. Espresso Shop's ten tests split into two shards:

-e numShards 2 -e shardIndex 0   ->  HelpLinkTest, 1 of RobotLoginTest       OK (2 tests)
-e numShards 2 -e shardIndex 1   ->  everything else                         OK (8 tests)

Two plus eight is ten, so nothing was lost or run twice, but the split is far from even. The runner assigns tests to shards by a hash of the test's name, not by duration or count. With a handful of tests, shards are lopsided; with hundreds, they even out. Two consequences:

  • Sharding pays off for large suites. For ten tests, one device is simpler.
  • Every shard must be able to run on its own. A test that only passes because another test ran before it in the same process will fail when the hash puts them on different devices, which is one more reason for the explicit state resets from chapter 5.

You shard a 400-test suite over 4 devices. Shard 2 finishes in 3 minutes and shard 0 takes 14. What explains it?

The runner's sharding balances names, not time. A few slow tests can end up in the same shard. If the imbalance hurts, split the slow tests off with an annotation and run them as their own job, or use more shards so the slow ones spread out.

You now have two files that run anywhere with adb. The next chapter hands them to TestingBot.