# Guide to Openlayer Webhooks: Features and Best Practices

[Openlayer](https://openlayer.com) tests and monitors AI systems. You define tests on your models, datasets and production inference pipelines, such as integrity checks with thresholds, and Openlayer runs them as suites against each new version or on live traffic. Its webhooks report when tests are created, changed or deleted and, most usefully, when a suite finishes a run, with counts of passing and failing tests. Teams use them to page on-call engineers when production tests start failing, kick off CI/CD steps when a suite completes, and keep test definitions in sync with their own systems.

## What are Openlayer webhooks?

Openlayer webhook subscriptions belong to a workspace and are managed by workspace admins. A subscription has an HTTPS URL and one or more event types, and Openlayer sends each event as a JSON `POST` with the `Openlayer-Webhooks/1.0` user agent.

Openlayer follows the Standard Webhooks specification and says any compatible library can verify its signatures. Each subscription has its own `whsec_` secret, shown once when you create it.

## Openlayer webhook features

| Feature | Details |
| --- | --- |
| Configuration | **Workspace Settings > Webhooks**, or the REST API. Admins only |
| Signature | Standard Webhooks HMAC-SHA256 |
| Signature header | `webhook-signature` |
| Secret | Per subscription, `whsec_...`, shown once |
| Body | JSON |
| Event routing | Body field `type` |
| Response window | `2xx` within 10 seconds |
| Retries | Up to 3, at about 5 seconds, 5 minutes and 30 minutes |
| History | Events and delivery records kept 90 days |
| User agent | `Openlayer-Webhooks/1.0` |

## Events and payloads

Every event has three top-level fields:

| Field | Meaning |
| --- | --- |
| `type` | The event type, for example `test.created`. This is what handlers switch on |
| `timestamp` | ISO 8601 (UTC) time at which the event was generated |
| `data` | Event-specific object: `data.test` for `test.*`, `data.tests` for `tests.result.updated` |
| `data.test.type` | The test's own category (for example `integrity`). Not the event type |

The `webhook-id` header stays the same across retries, so use it to deduplicate.

Openlayer has four event types:

| Event | Triggered When |
| --- | --- |
| `test.created` | A test is created. `data.test` contains the full test definition |
| `test.updated` | A test's configuration is updated. `data.test` has the same structure as `test.created`, reflecting the new state |
| `test.deleted` | A test is deleted. `data.test` contains only the test `id` |
| `tests.result.updated` | A test suite finishes a run. One event per run, summarising the results |

`tests.result.updated` is the one most integrations act on. Its `data` summarises a single run:

| Field | Meaning |
| --- | --- |
| `projectId` | The project the test suite belongs to |
| `projectVersionId` | The project version (commit) that was evaluated; may be `null` |
| `inferencePipelineId` | The inference pipeline that was evaluated, for monitoring runs; may be `null` |
| `total` | Total number of tests in the run |
| `passing` | Tests that passed |
| `failing` | Tests that failed |
| `skipped` | Tests that were skipped |
| `running` | Tests still running |

A development run has a `projectVersionId` (the commit evaluated), while a monitoring run has an `inferencePipelineId`. Use whichever is set to tell the two apart.

This is the `test.created` sample from the [webhook samples library](https://github.com/hookdeck/webhook-samples/blob/main/providers/openlayer/latest/test.created.json). Note that `data.test.type` is `integrity`, the kind of test, not the event type:

```json
{
  "type": "test.created",
  "timestamp": "2026-01-21T10:30:00Z",
  "data": {
    "test": {
      "id": "3fa85f64-5717-4562-b3fc-2c963f66afa6",
      "number": 1,
      "name": "No duplicate rows",
      "description": "This test checks for duplicate rows in the dataset.",
      "type": "integrity",
      "subtype": "duplicateRowCount",
      "dateCreated": "2026-01-21T10:30:00Z",
      "dateUpdated": "2026-01-21T10:30:00Z",
      "creatorId": "589ece63-49a2-41b4-98e1-10547761d4b0",
      "originProjectVersionId": "3fa85f64-5717-4562-b3fc-2c963f66afa6",
      "thresholds": [
        {
          "measurement": "duplicateRowCount",
          "insightName": "duplicateRowCount",
          "insightParameters": [],
          "operator": "<=",
          "value": 0
        }
      ],
      "evaluationWindow": 3600,
      "delayWindow": 0,
      "suggested": false,
      "archived": false
    }
  }
}

```

## Setting up Openlayer webhooks

In the dashboard, click the workspace name in the upper-left corner, choose **Workspace Settings**, open **Webhooks**, and add a webhook with a **URL**, one or more **Event types** and an optional **Description**. New subscriptions are enabled straight away.

With the API, authenticate with an API key as a bearer token:

```bash
curl -X POST "https://api.openlayer.com/v1/workspaces/{workspaceId}/webhooks" \
  -H "Authorization: Bearer $OPENLAYER_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://example.com/webhooks/openlayer",
    "description": "Production alerting",
    "eventTypes": ["test.created", "tests.result.updated"]
  }'

```

`url` (up to 2048 characters) and `eventTypes` are required; `description` (up to 500) and `enabled` (default `true`) are optional. List, get, update (`PUT`) and delete calls work on `.../webhooks/{webhookId}`, and `GET .../{webhookId}/deliveries` shows delivery attempts.

The create response, and the dashboard right after creation, show the secret once:

```json
{ "id": "3fa85f64-5717-4562-b3fc-2c963f66afa6", "secret": "whsec_MfKQ9r8GKYqrTwjUPD8ILPZIo2LaLaSw" }

```

Openlayer never returns it again, not even when listing or retrieving subscriptions. Store the whole value in `OPENLAYER_WEBHOOK_SECRET` immediately.

To route Openlayer through Hookdeck, create a source with the **Openlayer** type, paste the secret, and use the source URL as the subscription's `url`. The [Hookdeck CLI](/docs/cli) forwards it to a local server:

```bash
npx hookdeck-cli listen 3000 openlayer --path /webhooks/openlayer

```

Creating a test in the workspace is the quickest way to trigger a `test.created`.

## Securing Openlayer webhooks

Openlayer's docs give `signed_content = "{webhook-id}.{webhook-timestamp}.{raw_body}"`, keyed with the secret minus `whsec_`, base64-decoded. The base64 result, prefixed with `v1,`, is one of the space-delimited entries in `webhook-signature`, and the function below accepts any matching one:

```javascript
const crypto = require('node:crypto');

function decodeBase64(value) {
  if (typeof value !== 'string' || !/^[A-Za-z0-9+/]+={0,2}$/.test(value)) return null;
  const bytes = Buffer.from(value, 'base64');
  return bytes.length > 0 && bytes.toString('base64').replace(/=+$/, '') === value.replace(/=+$/, '')
    ? bytes : null;
}

function verifySignature(rawBody, headers, secret) {
  if (!Buffer.isBuffer(rawBody) || typeof secret !== 'string' || !secret) return false;
  const prefix = headers['webhook-id'] !== undefined ? 'webhook' : 'svix';
  const id = headers[`${prefix}-id`];
  const timestamp = headers[`${prefix}-timestamp`];
  const signature = headers[`${prefix}-signature`];
  if (typeof id !== 'string' || !id || typeof timestamp !== 'string' ||
      !/^[0-9]+$/.test(timestamp) || typeof signature !== 'string') return false;
  const key = decodeBase64(secret.replace(/^whsec_/, ''));
  if (!key) return false;
  const message = Buffer.concat([Buffer.from(`${id}.${timestamp}.`, 'utf8'), rawBody]);
  const candidates = signature.split(/\s+/).filter((item) => item.startsWith('v1,')).map((item) => item.slice(3));
  const expected = crypto.createHmac('sha256', key).update(message).digest();
  return candidates.some((candidate) => {
    const received = decodeBase64(candidate);
    if (!received) return false;
    return received.length === expected.length && crypto.timingSafeEqual(received, expected);
  });
}

```

Openlayer suggests rejecting a request whose timestamp is, for example, more than five minutes old. Its own retries end about half an hour after the first attempt, but Hookdeck replays and retries with the original `webhook-timestamp` and has no tolerance itself, so behind Hookdeck skip that check, as the verifier above does.

Test results often trigger CI jobs or pages, which take longer than ten seconds, so the receiver queues them:

```javascript
const express = require('express');

// Returns the parsed body when it is a JSON object, otherwise null.
function parseJsonObject(rawBody) {
  try {
    const value = JSON.parse(rawBody.toString('utf8'));
    return value && typeof value === 'object' && !Array.isArray(value) ? value : null;
  } catch {
    return null;
  }
}

// evalEvents.enqueue(webhookId, event) stores the event and resolves once it is durable.
function createOpenlayerReceiver(evalEvents) {
  const app = express();
  app.post('/webhooks/openlayer', express.raw({ type: '*/*' }), async (req, res) => {
    const secret = process.env.OPENLAYER_WEBHOOK_SECRET;
    if (!secret) return res.sendStatus(500);
    if (!verifySignature(req.body, req.headers, secret)) return res.sendStatus(401);

    const event = parseJsonObject(req.body);
    if (!event) return res.sendStatus(400); // signed, but not a JSON object
    try {
      await evalEvents.enqueue(req.headers['webhook-id'], event);
    } catch {
      return res.sendStatus(503);
    }
    return res.sendStatus(200);
  });
  return app;
}

```

A Hookdeck Openlayer source always answers Openlayer with `200`. A request with a mismatched signature is logged as failed verification and isn't delivered.

> **Make Openlayer webhooks production-ready.** [Hookdeck Event Gateway](/event-gateway) holds test results past Openlayer's half-hour retry window and can route failing runs to your alerting tools.

## Openlayer webhook limitations and pain points

### Retries stop after about 30 minutes

**The Problem:** A failing-tests alert sent during a short outage is lost.

**Why It Happens:** Openlayer makes the first attempt plus three retries, at about 5 seconds, 5 minutes and 30 minutes, then marks the event failed.

**Workarounds:**

* Check `GET .../{webhookId}/deliveries` after an outage, and query test results through the API.

**How Hookdeck Can Help:** Hookdeck accepts each result immediately and [retries](/docs/retries) your service on a longer schedule that you set.

### The secret can't be recovered

**The Problem:** A lost secret means you can't verify deliveries.

**Why It Happens:** Openlayer returns the secret only on create. Listing and retrieving never include it.

**Workarounds:**

* Delete the subscription and create a new one, then update the stored secret.

**How Hookdeck Can Help:** Storing the secret on a Hookdeck source keeps it in one place, and if you recreate the subscription, you only change it there.

### Events can arrive out of order

**The Problem:** A `test.updated` is applied after the `test.deleted` for the same test.

**Why It Happens:** A retried event can arrive after later ones.

**Workarounds:**

* Order by the body's `timestamp`, and treat deletion as final for a test `id`.

**How Hookdeck Can Help:** Hookdeck's event history shows the order in which events actually arrived, which makes this easy to diagnose.

### Admins only

**The Problem:** Engineers who own the alerting integration can't set it up.

**Why It Happens:** Only workspace admins can create and manage webhooks.

**Workarounds:**

* Have an admin create the subscription through the API with a script you maintain.

**How Hookdeck Can Help:** Point one subscription at a Hookdeck source, and your team can add destinations and [filters](/docs/filters) without touching Openlayer settings.

## Best practices

### Alert on `failing`, not on every run

`tests.result.updated` fires for every run. Page someone when `failing` is above zero for a monitoring pipeline, and stay quiet otherwise.

### Report development runs back to CI

On a development run, `projectVersionId` identifies the version that was evaluated. Use it to attach the pass and fail counts to the matching build.

### Treat `test.deleted` as an ID only

`data.test` holds only the `id` on deletion. Remove the test from your copy by ID.

### Deduplicate on `webhook-id`

Retries reuse it, so record it with each action; see the [webhook idempotency guide](/webhooks/guides/implement-webhook-idempotency).

## Conclusion

Openlayer's four events are signed with Standard Webhooks and recorded in a delivery log you can query. Save the secret when it's shown, because it won't be shown again.

[Hookdeck Event Gateway](https://hookdeck.com) adds longer retries and routing so failing AI tests reach the right people. [Sign up for free](https://dashboard.hookdeck.com/signup).