Re-run Failed Tests
When a few tests of a large suite fail, running only those again is faster than repeating the whole suite. How you do that depends on who runs the tests:
- Espresso, XCUITest and Maestro: TestingBot runs the test suite on the device, so TestingBot can start a new run with only the failed tests. Use the dashboard or the API.
- Selenium and Appium: your own test runner (in your CI) starts every session, so a re-run has to start there too. TestingBot gives you the list of failed tests of a build to feed back into your runner.
Espresso and XCUITest
A re-run starts a new run on the same device (the same capabilities) with the same app and test package. It does not upload anything again. The new run shows up in the same project, next to the run it repeats, and links back to it.
From the dashboard
Open a failed run from your Espresso or XCUITest projects and click Re-run failed. The button is shown on finished runs that have failed tests, to team members who can start tests. You land on the new run.
With the API
To re-run every failed run of a project in one call, for example from a CI step after the first attempt, use the rerun-failed endpoint with the project ID the run endpoint returned:
Espresso:
curl -u api_key:api_secret \
-X POST "https://api.testingbot.com/v1/app-automate/espresso/:id/rerun-failed"
XCUITest:
curl -u api_key:api_secret \
-X POST "https://api.testingbot.com/v1/app-automate/xcuitest/:id/rerun-failed"
The response lists one new run per failed run:
{
"success": true,
"id": 4211,
"runs": [
{
"id": 90311,
"rerun_of": 90297,
"capabilities": { "platformName": "Android", "deviceName": "Pixel 8", "version": "14" },
"tests": ["com.example.LoginTest#logsInWithEmail", "com.example.CartTest#removesItem"]
}
],
"priority": "normal"
}
Poll each new run with GET /v1/app-automate/espresso/:project_id/:run_id (or xcuitest) until it is done. The project status (GET /v1/app-automate/espresso/:id) keeps reporting the earlier failed runs as well, so check the new runs rather than the project.
Optional parameters:
-
run_ids: only re-run these runs of the project. By default every failed run that has not been re-run yet is re-run, so calling the endpoint again after a re-run only repeats what still fails, and answers409once nothing does. -
espressoOptions(Espresso) oroptions(XCUITest): the run options to use, in the same format as when starting a run. Without it, each run is re-run with the options it was started with. -
priority:high,normalorlow, as for a new run. -
metadata: CI / CD metadata for the new runs. Without it, they keep the metadata of the runs they repeat.
The call is refused with HTTP 409 while any run of the project is still queued or running, or when the project has no failed tests. See the API reference.
Which tests run again
-
Espresso: the failed tests are passed as a class filter of
Class#methodentries. A test whose method name cannot be used in a filter (for example a parameterized test) re-runs its whole class. -
XCUITest: the failed tests are passed as
onlyTestingentries inClass/methodform. - Other test filters of the original run (such as
package,annotation, sharding, orskipTesting) are left out, since they would narrow the list of failed tests further. - A run that failed before it reported any tests (for example because the app could not be installed) runs its whole suite again, with its original filters. The same happens when a failed test cannot be selected by name, or when more than 500 tests failed in one run.
Maestro
On a Maestro run page, Retry failed flows runs every failed flow again in a new session on the same device, and each failed flow has its own Retry this flow. From the command line (testingbotctl 1.5.0 or later), testingbot rerun --id PROJECT_ID re-runs every failed flow of a project, and --flow picks flows by name.
Selenium and Appium builds
Selenium and Appium sessions are started by your own test code, so TestingBot cannot start them again by itself. Instead, give every test a name and every CI run its own build name, then ask the Builds API for the failed tests of that build:
curl -u api_key:api_secret \
"https://api.testingbot.com/v1/builds/build-1234?status=failed&count=500&skip_fields=logs,thumbs"
The build can be addressed by its name or by its numeric ID. Each entry in data has the test name, session_id, browser, os and status_message; meta.total counts the failed tests, so page with offset when there are more than count. To print just the names, for example:
curl -s -u api_key:api_secret \
"https://api.testingbot.com/v1/builds/build-1234?status=failed&count=500&skip_fields=logs,thumbs" \
| jq -r '.data[].name'
Pass those names to your test runner's own filter to run them again, in a new build. status also accepts passed, and unknown for tests whose result was never set. A test counts as failed once its result is set to failed, either from your test code or through the API.