Skip to main content

Create incident.io Alerts from Failed Tests

With the incident.io webhook template, TestingBot fires an alert in incident.io the moment a test fails. From there, incident.io takes over: the alert is grouped and routed by your alert configuration, can page the on-call engineer through an escalation path, and can create an incident with the test name, browser, platform and a direct link to the test details page.

The template uses the incident.io Alert Events API through an HTTP alert source: you only need the alert source URL and its secret, no incident.io user account credentials are involved.

Prerequisites

You need an HTTP alert source in incident.io:

  1. In incident.io, go to Alerts → Configure and create a new alert source of type HTTP. Pick the Default payload option, name it for example TestingBot.
  2. incident.io shows the endpoint URL for the source, in the form https://api.incident.io/v2/alert_events/http/<CONFIG_ID>. Copy it.
  3. Copy the secret (token) that incident.io generated for the source. It is used as a Bearer token when TestingBot delivers the alert.
  4. Connect the alert source to an alert route so incoming alerts create escalations or incidents. Without a route, alerts arrive but nothing is paged.

Set up the webhook in TestingBot

  1. Log in to the TestingBot member area and open the Webhooks page under Integrations.
  2. Click Create webhook and give it a name, for example incident.io alerts.
  3. Under Start from a template, pick the incident.io tile. This prefills the POST method, the Authorization header and the payload below.
  4. Replace YOUR_ALERT_SOURCE_CONFIG_ID in the URL with the id from the endpoint URL you copied from incident.io (or paste the full URL).
  5. In the Headers tab, replace YOUR_ALERT_SOURCE_SECRET with the secret of your alert source, keeping the Bearer prefix.
  6. Set the trigger to failures only, so an alert fires only when a test fails. Optionally narrow it down with a test type or a name filter such as regression_*.
  7. Click Create webhook at the bottom of the form to save the configuration.

Template payload

The incident.io template sends this JSON body. The {{...}} tokens are substituted with test values at delivery time and are documented on the built-in variables page. 

{
  "title": "Test {{TEST_STATUS}}: {{TEST_NAME}}",
  "description": "Test {{TEST_NAME}} finished with status {{TEST_STATUS}} on {{PLATFORM}}. View it at {{TEST_URL}}",
  "status": "firing",
  "deduplication_key": "{{TEST_ID}}",
  "source_url": "{{TEST_URL}}",
  "metadata": {
    "browser": "{{BROWSER_DISPLAY_NAME}}",
    "platform": "{{PLATFORM}}",
    "build": "{{BUILD_NAME}}",
    "duration_sec": {{TEST_DURATION}},
    "tags": {{TEST_TAGS}}
  }
}

The alert title contains the test result and name, the source_url opens the test on TestingBot, and the metadata object carries the browser, platform, build, duration and tags. In incident.io you can map metadata fields to alert attributes and priorities, for example routing by tag or team.

Everything stays editable. One useful tweak: change the deduplication_key to {{TEST_NAME}} so repeated failures of the same test collapse into a single alert instead of paging again for every run.

Auto-resolving alerts

The template always sends "status": "firing", so alerts are opened but never closed by TestingBot. If you want an alert to resolve automatically when the test passes again, use a Custom HTTP alert source in incident.io instead of a Default one:

  1. In the TestingBot webhook, change the payload to "status": "{{TEST_STATUS}}", set the deduplication_key to {{TEST_NAME}} and set the trigger to all tests instead of failures only.
  2. In incident.io, create the HTTP alert source with the Custom payload option and use a transform expression that maps the TestingBot status onto the alert status: 
return {
  title: $.title,
  description: $.description,
  status: $.status === "PASSED" ? "resolved" : "firing",
  source_url: $.source_url,
  metadata: $.metadata,
};

Set the deduplication key path of the source to deduplication_key. A failing test now opens an alert, and the next passing run of the same test resolves it.

Test it

Click Test Webhook in the form to send a sample event with placeholder test values. The response panel shows the HTTP status, duration, headers and body returned by incident.io; a successful request returns a 202 Accepted status with "status": "success" in the body, and a new alert appears on the alert source in incident.io.

If no alert shows up, check these common causes:

  • The Authorization header is wrong or still set to the YOUR_ALERT_SOURCE_SECRET placeholder; incident.io responds with a 401 status in that case.
  • The URL still contains the YOUR_ALERT_SOURCE_CONFIG_ID placeholder, or the id belongs to a different alert source.
  • The alert arrived but is not connected to an alert route, so no escalation or incident was created. Check the alert source's recent events in incident.io.
Was this page helpful?
Last updated
TestingBot Docs