Webhooks Core Concepts

When webhooks fire

A webhook fires after each completed test that matches its trigger settings. You configure three filters when creating or editing a webhook at /members/integrations/webhooks:

• Result filter: fire on failures only, or on all tests regardless of their result.
• Test type: include all tests, only automated browser tests or only real device tests.
• Name filter: an optional wildcard pattern such as regression_*. The webhook only fires when the pattern matches the test name or the build name.

Requests may arrive with a short delay after a test completes.

Default and custom payloads

Each webhook sends one of two payload types, selected with the Default | Custom toggle in the webhook form:

• Default: a fixed JSON document with all the test details, documented in the reference below. This is the easiest option when your own endpoint consumes the request.
• Custom: a JSON body that you write yourself, using built-in variables such as {{TEST_NAME}} and {{TEST_STATUS}}. Use this to match the format an external service expects, or pick one of the templates to prefill a custom payload for a popular service.

Default payload reference

When you do not configure a custom payload, your webhook URL receives a POST request in JSON format with the following payload:

errors = [
  {
    message: "The error message",
    test_name: "The name of the sub test, if any. For example a Flow name when testing with Maestro"
  }
]

Authorization, headers and parameters

The Authorization, Headers and Params tabs in the webhook form control how the HTTP request is sent. The tabs appear when the Custom payload mode is selected; with the Default payload, only the Authorization fields are shown:

• Authorization: choose No Auth, Basic Auth (username and password) or a Bearer token. The credentials are added to the Authorization header of every request.
• Headers: add custom HTTP headers, for example an API key header that the receiving service requires.
• Params: add query parameters that are appended to the webhook URL.

Header values, query parameter values and the URL itself support built-in variables, so you can include test details such as {{TEST_ID}} outside of the payload body.

Delivery and security

• Delivery is best effort and is not retried: make sure your endpoint is available and responds quickly.
• Your endpoint is expected to return a 2xx status code.
• Requests time out after 90 seconds.
• Redirects are not followed: point the webhook directly at the final URL.
• Every delivery is signed with an HMAC-SHA256 signature, so your endpoint can verify that the request really comes from TestingBot. You can additionally configure authorization as a second factor.
• Webhook URLs that resolve to private or internal addresses are refused; the endpoint must be reachable on the public internet.

Verify deliveries

Each webhook has a signing secret (it starts with whsec_), shown on the webhook's edit page in the members area, where you can also rotate it. TestingBot signs every delivery with it and adds two headers to the request:

To verify a delivery on your server:

1. Read the raw request body, before any JSON parsing or re-serialization. A single re-encoded space breaks the signature.
2. Reject the request when the timestamp is older than 5 minutes; this blocks replayed deliveries.
3. Compute the HMAC-SHA256 of "<timestamp>.<raw body>" with your signing secret and prefix it with sha256=.
4. Compare it with the X-TestingBot-Signature header using a constant-time comparison, never a plain string equality.

import javax.crypto.Mac;
import javax.crypto.spec.SecretKeySpec;
import java.nio.charset.StandardCharsets;
import java.security.MessageDigest;

public class TestingBotWebhook {
  public static boolean verifySignature(String rawBody, String timestamp,
                                        String signature, String secret) throws Exception {
    if (Long.parseLong(timestamp) < (System.currentTimeMillis() / 1000) - 300) {
      return false;
    }

    Mac mac = Mac.getInstance("HmacSHA256");
    mac.init(new SecretKeySpec(secret.getBytes(StandardCharsets.UTF_8), "HmacSHA256"));
    byte[] digest = mac.doFinal((timestamp + "." + rawBody).getBytes(StandardCharsets.UTF_8));

    StringBuilder hex = new StringBuilder();
    for (byte b : digest) {
      hex.append(String.format("%02x", b));
    }
    String expected = "sha256=" + hex;

    // MessageDigest.isEqual is a constant-time comparison.
    return MessageDigest.isEqual(
      expected.getBytes(StandardCharsets.UTF_8),
      (signature == null ? "" : signature).getBytes(StandardCharsets.UTF_8));
  }
}

const crypto = require('crypto');

function verifyTestingBotSignature(rawBody, timestamp, signature, secret) {
  if (Number(timestamp) < Math.floor(Date.now() / 1000) - 300) return false;

  const expected = 'sha256=' + crypto.createHmac('sha256', secret)
    .update(`${timestamp}.${rawBody}`)
    .digest('hex');
  const a = Buffer.from(expected);
  const b = Buffer.from(String(signature || ''));
  return a.length === b.length && crypto.timingSafeEqual(a, b);
}

// Express needs the raw body, for example:
// app.use(express.json({ verify: (req, res, buf) => { req.rawBody = buf.toString(); } }));

require 'openssl'
require 'rack/utils'

def verify_testingbot_signature(raw_body, timestamp, signature, secret)
  return false if timestamp.to_i < Time.now.to_i - 300

  expected = 'sha256=' + OpenSSL::HMAC.hexdigest('SHA256', secret, "#{timestamp}.#{raw_body}")
  Rack::Utils.secure_compare(expected, signature.to_s)
end

# Rails example:
# verify_testingbot_signature(
#   request.raw_post,
#   request.headers['X-TestingBot-Timestamp'],
#   request.headers['X-TestingBot-Signature'],
#   ENV['TESTINGBOT_WEBHOOK_SECRET']
# )

import hashlib
import hmac
import time

def verify_testingbot_signature(raw_body, timestamp, signature, secret):
    if int(timestamp) < int(time.time()) - 300:
        return False

    signed = f"{timestamp}.{raw_body}".encode()
    expected = "sha256=" + hmac.new(secret.encode(), signed, hashlib.sha256).hexdigest()
    return hmac.compare_digest(expected, signature or "")

using System;
using System.Security.Cryptography;
using System.Text;

public static class TestingBotWebhook
{
    public static bool VerifySignature(string rawBody, string timestamp,
                                       string signature, string secret)
    {
        if (long.Parse(timestamp) < DateTimeOffset.UtcNow.ToUnixTimeSeconds() - 300)
        {
            return false;
        }

        using var hmac = new HMACSHA256(Encoding.UTF8.GetBytes(secret));
        var digest = hmac.ComputeHash(Encoding.UTF8.GetBytes($"{timestamp}.{rawBody}"));
        var expected = "sha256=" + Convert.ToHexString(digest).ToLowerInvariant();

        // FixedTimeEquals is a constant-time comparison.
        return CryptographicOperations.FixedTimeEquals(
            Encoding.UTF8.GetBytes(expected),
            Encoding.UTF8.GetBytes(signature ?? string.Empty));
    }
}

// ASP.NET Core needs the raw body, for example:
// using var reader = new StreamReader(Request.Body);
// var rawBody = await reader.ReadToEndAsync();

The Test Webhook button signs its request with the webhook's real secret when you test a saved webhook, so you can exercise your verification code from the edit page before any real test fires. After rotating the secret, update your receiver immediately: deliveries signed with the old secret no longer validate.

Limits

• You can create up to 5 webhooks per team.
• A custom payload template can be at most 64,000 characters.
• The response shown in the Test Webhook panel is truncated to 10,000 characters.

Related

• Built-in variables: every variable you can use in custom payloads, URLs, headers and parameters.
• Integrations: ready-made templates for Slack, Teams, Discord, PagerDuty, OpsGenie and Jira.
• Quick Start: create and test your first webhook step by step.