Exit course
Chapter 6 of 10 Intermediate 7 min read 2 quick checks Updated

Test Isolation, Orchestrator and the Robot Pattern

On this page
  1. The failure
  2. Fix 1: Android Test Orchestrator
  3. Fix 2: reset state in the test
  4. The robot pattern

Every test so far passes when you run it alone. Run the whole suite in one go, and two product list tests fail. This chapter finds out why, fixes it in two different ways (and explains why you want the second one even when you have the first), and then makes the tests easier to read with the robot pattern.

The failure#

Without any isolation in place, a full run of the suite on the emulator produced:

Error in basketStartsEmpty(com.example.shop.ProductListTest):
Expected: ... is "Basket: empty"
     Got: view.getText() was "Basket: Product 1, Product 2"

Error in addsAnItemFarDownTheList(com.example.shop.ProductListTest):
Expected: ... is "Basket: Product 42"
     Got: view.getText() was "Basket: Product 1, Product 2, Product 42"

addsTwoItems ran first and put two products in the basket. Espresso Shop saves the basket in SharedPreferences so that it survives app restarts, and it survived into the next test too.

The reason is how instrumentation works. AndroidJUnitRunner runs every test in one app process, with one data directory. ActivityScenarioRule gives each test a new activity, but files, databases, SharedPreferences, singletons and static fields all carry over. The tests are only independent if something makes them so. JUnit 4 does not promise a useful order either, so a suite like this can pass for months and break when someone adds an unrelated test that changes the order.

Each test class uses ActivityScenarioRule. A singleton Session.currentUser set by a login test is still set when a later test opens the login screen. Why?

The rule launches a fresh activity for every test and closes it afterwards, but the process keeps running, so singletons, static fields and stored data persist. Only a new process (Orchestrator) or an explicit reset clears them.

Fix 1: Android Test Orchestrator#

Android Test Orchestrator is a small app from AndroidX Test that runs each test in its own invocation of the instrumentation, which means its own app process. With the clearPackageData argument it also wipes the app's data between tests. Enable it in app/build.gradle.kts:

android {
    defaultConfig {
        testInstrumentationRunner = "androidx.test.runner.AndroidJUnitRunner"
        testInstrumentationRunnerArguments["clearPackageData"] = "true"
    }

    testOptions {
        execution = "ANDROIDX_TEST_ORCHESTRATOR"
        animationsDisabled = true
    }
}

dependencies {
    androidTestUtil("androidx.test:orchestrator:1.6.1")
}

androidTestUtil makes Gradle install the Orchestrator on the device next to your two APKs before the run. With these lines, ./gradlew connectedDebugAndroidTest ran all ten tests green on the emulator.

Orchestrator has a second benefit: if one test crashes the app process, only that test fails. Without it, a crash ends the whole instrumentation run, and every test after it is missing from the report.

The costs: every test pays for a process start, so suites get slower, and Orchestrator is a feature of how the tests are launched, not of the test APK. Gradle and Android Studio launch tests through it; adb shell am instrument and other runners that start AndroidJUnitRunner directly do not. TestingBot's Espresso documentation describes running your suite with AndroidJUnitRunner, and the clearPackageData argument from build.gradle.kts is not part of either APK. Do not count on Orchestrator anywhere outside your own Gradle run.

Fix 2: reset state in the test#

The portable fix is for each test to set up the state it needs. For the basket, that means emptying it before the activity starts. The first attempt that comes to mind does not work:

@get:Rule
val activityRule = ActivityScenarioRule<ProductsActivity>(intent)

@Before
fun emptyBasket() {
    Basket(ApplicationProvider.getApplicationContext()).clear()
}

basketStartsEmpty still failed with "Basket: Product 1, Product 2" in this course's run. JUnit rules wrap @Before, so the activity had already started, read the old basket and drawn it before emptyBasket() ran. The data was cleared, but the screen was already showing the stale value.

The reset has to happen before the activity launches, which means it has to be a rule itself, ordered outside the activity rule:

class EmptyBasketRule : ExternalResource() {
    override fun before() {
        Basket(ApplicationProvider.getApplicationContext()).clear()
    }
}
// Outer rule: empty the basket BEFORE the activity starts and reads it.
@get:Rule
val rules: RuleChain = RuleChain
    .outerRule(EmptyBasketRule())
    .around(ActivityScenarioRule<ProductsActivity>(intent))

With that rule, the whole suite passed twice in a row through plain adb shell am instrument, without Orchestrator: OK (10 tests) both times. That is the setup that matters for cloud runs, and it also keeps the suite honest locally.

Use both: the explicit resets make every test correct wherever it runs, and Orchestrator catches the state you did not know about and isolates crashes on your own machine and CI emulators.

A test seeds a fake user into the app's database in @Before, but the screen under test, launched by ActivityScenarioRule, shows "No user". What is the fix?

The activity rule launches the screen before @Before runs, so the screen read the database before the seed existed. Seeding later in the test method is even later. Orchestrator would clear data, not add it. Seed in an outer rule (RuleChain), or drop the rule and call ActivityScenario.launch after seeding.

The robot pattern#

Espresso statements are precise but noisy. A test that logs in, adds three products and checks the basket is twenty lines of onView(withId(...)).perform(...), and the intent gets lost. The robot pattern is the Android community's name for page objects: one class per screen that knows the screen's views, and tests that only talk to the robots. If you know page objects from web testing, the Page Object Model course covers the same idea and its pitfalls in more depth.

A login robot for Espresso Shop:

fun login(block: LoginRobot.() -> Unit) = LoginRobot().apply(block)

class LoginRobot {
    fun username(value: String) {
        onView(withId(R.id.username)).perform(replaceText(value))
    }

    fun password(value: String) {
        onView(withId(R.id.password)).perform(replaceText(value), closeSoftKeyboard())
    }

    infix fun submit(then: ProductsRobot.() -> Unit): ProductsRobot {
        onView(withId(R.id.login)).perform(click())
        return ProductsRobot().apply(then)
    }

    fun submitExpectingError(message: Int) {
        onView(withId(R.id.login)).perform(click())
        onView(withId(R.id.error)).check(matches(withText(message)))
    }
}

class ProductsRobot {
    fun isWelcoming(user: String) {
        onView(withId(R.id.welcome)).check(matches(withText("Welcome, $user")))
    }
}

The tests read like the scenario:

@Test
fun aliceCanLogIn() {
    login {
        username("alice")
        password("secret")
    } submit {
        isWelcoming("alice")
    }
}

@Test
fun wrongPasswordIsRejected() {
    login {
        username("alice")
        password("wrong")
        submitExpectingError(R.string.login_failed)
    }
}

The Kotlin features doing the work: login { ... } takes a lambda with LoginRobot as its receiver, so username(...) inside it calls the robot; infix fun submit lets the call be written login { } submit { }; and submit returns the next screen's robot, so a test can only call checks that belong to the screen it is on.

Guidelines that keep robots useful:

  • One robot per screen, holding that screen's matchers. When a layout changes, one file changes.
  • Robots do things and check things; tests decide what. Keep if statements and test data out of robots.
  • Keep the synchronisation out of robots. The IdlingResource registration stays in the test class (@Before/@After), so a robot works the same in every test.
  • Do not hide every line. A robot method for a single onView call that is used once adds a layer without adding meaning.

The robot test class registers AuthService.idlingResource exactly like LoginTest does. With it, the suite has ten tests across five classes, all green locally, with and without Orchestrator. The next step is to package them for somewhere other than your machine.