Gareth Wilson Gareth Wilson

Guide to Openlayer Webhooks: Features and Best Practices

Published

Openlayer tests and monitors AI systems. You define tests on your models, datasets and production inference pipelines, such as integrity checks with thresholds, and Openlayer runs them as suites against each new version or on live traffic. Its webhooks report when tests are created, changed or deleted and, most usefully, when a suite finishes a run, with counts of passing and failing tests. Teams use them to page on-call engineers when production tests start failing, kick off CI/CD steps when a suite completes, and keep test definitions in sync with their own systems.

What are Openlayer webhooks?

Openlayer webhook subscriptions belong to a workspace and are managed by workspace admins. A subscription has an HTTPS URL and one or more event types, and Openlayer sends each event as a JSON POST with the Openlayer-Webhooks/1.0 user agent.

Openlayer follows the Standard Webhooks specification and says any compatible library can verify its signatures. Each subscription has its own whsec_ secret, shown once when you create it.

Openlayer webhook features

FeatureDetails
ConfigurationWorkspace Settings > Webhooks, or the REST API. Admins only
SignatureStandard Webhooks HMAC-SHA256
Signature headerwebhook-signature
SecretPer subscription, whsec_..., shown once
BodyJSON
Event routingBody field type
Response window2xx within 10 seconds
RetriesUp to 3, at about 5 seconds, 5 minutes and 30 minutes
HistoryEvents and delivery records kept 90 days
User agentOpenlayer-Webhooks/1.0

Events and payloads

Every event has three top-level fields:

FieldMeaning
typeThe event type, for example test.created. This is what handlers switch on
timestampISO 8601 (UTC) time at which the event was generated
dataEvent-specific object: data.test for test.*, data.tests for tests.result.updated
data.test.typeThe test's own category (for example integrity). Not the event type

The webhook-id header stays the same across retries, so use it to deduplicate.

Openlayer has four event types:

EventTriggered When
test.createdA test is created. data.test contains the full test definition
test.updatedA test's configuration is updated. data.test has the same structure as test.created, reflecting the new state
test.deletedA test is deleted. data.test contains only the test id
tests.result.updatedA test suite finishes a run. One event per run, summarising the results

tests.result.updated is the one most integrations act on. Its data summarises a single run:

FieldMeaning
projectIdThe project the test suite belongs to
projectVersionIdThe project version (commit) that was evaluated; may be null
inferencePipelineIdThe inference pipeline that was evaluated, for monitoring runs; may be null
totalTotal number of tests in the run
passingTests that passed
failingTests that failed
skippedTests that were skipped
runningTests still running

A development run has a projectVersionId (the commit evaluated), while a monitoring run has an inferencePipelineId. Use whichever is set to tell the two apart.

This is the test.created sample from the webhook samples library. Note that data.test.type is integrity, the kind of test, not the event type:

{
  "type": "test.created",
  "timestamp": "2026-01-21T10:30:00Z",
  "data": {
    "test": {
      "id": "3fa85f64-5717-4562-b3fc-2c963f66afa6",
      "number": 1,
      "name": "No duplicate rows",
      "description": "This test checks for duplicate rows in the dataset.",
      "type": "integrity",
      "subtype": "duplicateRowCount",
      "dateCreated": "2026-01-21T10:30:00Z",
      "dateUpdated": "2026-01-21T10:30:00Z",
      "creatorId": "589ece63-49a2-41b4-98e1-10547761d4b0",
      "originProjectVersionId": "3fa85f64-5717-4562-b3fc-2c963f66afa6",
      "thresholds": [
        {
          "measurement": "duplicateRowCount",
          "insightName": "duplicateRowCount",
          "insightParameters": [],
          "operator": "<=",
          "value": 0
        }
      ],
      "evaluationWindow": 3600,
      "delayWindow": 0,
      "suggested": false,
      "archived": false
    }
  }
}

Setting up Openlayer webhooks

In the dashboard, click the workspace name in the upper-left corner, choose Workspace Settings, open Webhooks, and add a webhook with a URL, one or more Event types and an optional Description. New subscriptions are enabled straight away.

With the API, authenticate with an API key as a bearer token:

curl -X POST "https://api.openlayer.com/v1/workspaces/{workspaceId}/webhooks" \
  -H "Authorization: Bearer $OPENLAYER_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://example.com/webhooks/openlayer",
    "description": "Production alerting",
    "eventTypes": ["test.created", "tests.result.updated"]
  }'

url (up to 2048 characters) and eventTypes are required; description (up to 500) and enabled (default true) are optional. List, get, update (PUT) and delete calls work on .../webhooks/{webhookId}, and GET .../{webhookId}/deliveries shows delivery attempts.

The create response, and the dashboard right after creation, show the secret once:

{ "id": "3fa85f64-5717-4562-b3fc-2c963f66afa6", "secret": "whsec_MfKQ9r8GKYqrTwjUPD8ILPZIo2LaLaSw" }

Openlayer never returns it again, not even when listing or retrieving subscriptions. Store the whole value in OPENLAYER_WEBHOOK_SECRET immediately.

To route Openlayer through Hookdeck, create a source with the Openlayer type, paste the secret, and use the source URL as the subscription's url. The Hookdeck CLI forwards it to a local server:

npx hookdeck-cli listen 3000 openlayer --path /webhooks/openlayer

Creating a test in the workspace is the quickest way to trigger a test.created.

Securing Openlayer webhooks

Openlayer's docs give signed_content = "{webhook-id}.{webhook-timestamp}.{raw_body}", keyed with the secret minus whsec_, base64-decoded. The base64 result, prefixed with v1,, is one of the space-delimited entries in webhook-signature, and the function below accepts any matching one:

const crypto = require('node:crypto');

function decodeBase64(value) {
  if (typeof value !== 'string' || !/^[A-Za-z0-9+/]+={0,2}$/.test(value)) return null;
  const bytes = Buffer.from(value, 'base64');
  return bytes.length > 0 && bytes.toString('base64').replace(/=+$/, '') === value.replace(/=+$/, '')
    ? bytes : null;
}

function verifySignature(rawBody, headers, secret) {
  if (!Buffer.isBuffer(rawBody) || typeof secret !== 'string' || !secret) return false;
  const prefix = headers['webhook-id'] !== undefined ? 'webhook' : 'svix';
  const id = headers[`${prefix}-id`];
  const timestamp = headers[`${prefix}-timestamp`];
  const signature = headers[`${prefix}-signature`];
  if (typeof id !== 'string' || !id || typeof timestamp !== 'string' ||
      !/^[0-9]+$/.test(timestamp) || typeof signature !== 'string') return false;
  const key = decodeBase64(secret.replace(/^whsec_/, ''));
  if (!key) return false;
  const message = Buffer.concat([Buffer.from(`${id}.${timestamp}.`, 'utf8'), rawBody]);
  const candidates = signature.split(/\s+/).filter((item) => item.startsWith('v1,')).map((item) => item.slice(3));
  const expected = crypto.createHmac('sha256', key).update(message).digest();
  return candidates.some((candidate) => {
    const received = decodeBase64(candidate);
    if (!received) return false;
    return received.length === expected.length && crypto.timingSafeEqual(received, expected);
  });
}

Openlayer suggests rejecting a request whose timestamp is, for example, more than five minutes old. Its own retries end about half an hour after the first attempt, but Hookdeck replays and retries with the original webhook-timestamp and has no tolerance itself, so behind Hookdeck skip that check, as the verifier above does.

Test results often trigger CI jobs or pages, which take longer than ten seconds, so the receiver queues them:

const express = require('express');

// Returns the parsed body when it is a JSON object, otherwise null.
function parseJsonObject(rawBody) {
  try {
    const value = JSON.parse(rawBody.toString('utf8'));
    return value && typeof value === 'object' && !Array.isArray(value) ? value : null;
  } catch {
    return null;
  }
}

// evalEvents.enqueue(webhookId, event) stores the event and resolves once it is durable.
function createOpenlayerReceiver(evalEvents) {
  const app = express();
  app.post('/webhooks/openlayer', express.raw({ type: '*/*' }), async (req, res) => {
    const secret = process.env.OPENLAYER_WEBHOOK_SECRET;
    if (!secret) return res.sendStatus(500);
    if (!verifySignature(req.body, req.headers, secret)) return res.sendStatus(401);

    const event = parseJsonObject(req.body);
    if (!event) return res.sendStatus(400); // signed, but not a JSON object
    try {
      await evalEvents.enqueue(req.headers['webhook-id'], event);
    } catch {
      return res.sendStatus(503);
    }
    return res.sendStatus(200);
  });
  return app;
}

A Hookdeck Openlayer source always answers Openlayer with 200. A request with a mismatched signature is logged as failed verification and isn't delivered.

Make Openlayer webhooks production-ready. Hookdeck Event Gateway holds test results past Openlayer's half-hour retry window and can route failing runs to your alerting tools.

Openlayer webhook limitations and pain points

Retries stop after about 30 minutes

The Problem: A failing-tests alert sent during a short outage is lost.

Why It Happens: Openlayer makes the first attempt plus three retries, at about 5 seconds, 5 minutes and 30 minutes, then marks the event failed.

Workarounds:

  • Check GET .../{webhookId}/deliveries after an outage, and query test results through the API.

How Hookdeck Can Help: Hookdeck accepts each result immediately and retries your service on a longer schedule that you set.

The secret can't be recovered

The Problem: A lost secret means you can't verify deliveries.

Why It Happens: Openlayer returns the secret only on create. Listing and retrieving never include it.

Workarounds:

  • Delete the subscription and create a new one, then update the stored secret.

How Hookdeck Can Help: Storing the secret on a Hookdeck source keeps it in one place, and if you recreate the subscription, you only change it there.

Events can arrive out of order

The Problem: A test.updated is applied after the test.deleted for the same test.

Why It Happens: A retried event can arrive after later ones.

Workarounds:

  • Order by the body's timestamp, and treat deletion as final for a test id.

How Hookdeck Can Help: Hookdeck's event history shows the order in which events actually arrived, which makes this easy to diagnose.

Admins only

The Problem: Engineers who own the alerting integration can't set it up.

Why It Happens: Only workspace admins can create and manage webhooks.

Workarounds:

  • Have an admin create the subscription through the API with a script you maintain.

How Hookdeck Can Help: Point one subscription at a Hookdeck source, and your team can add destinations and filters without touching Openlayer settings.

Best practices

Alert on failing, not on every run

tests.result.updated fires for every run. Page someone when failing is above zero for a monitoring pipeline, and stay quiet otherwise.

Report development runs back to CI

On a development run, projectVersionId identifies the version that was evaluated. Use it to attach the pass and fail counts to the matching build.

Treat test.deleted as an ID only

data.test holds only the id on deletion. Remove the test from your copy by ID.

Deduplicate on webhook-id

Retries reuse it, so record it with each action; see the webhook idempotency guide.

Conclusion

Openlayer's four events are signed with Standard Webhooks and recorded in a delivery log you can query. Save the secret when it's shown, because it won't be shown again.

Hookdeck Event Gateway adds longer retries and routing so failing AI tests reach the right people. Sign up for free.

Gareth Wilson

Gareth Wilson

Product Marketing

Multi-time founding marketer, Gareth is PMM at Hookdeck and author of the newsletter, Community Inc.