Openlayer Webhooks Skill

Openlayer is an AI evaluation and observability platform. Workspace webhooks POST a signed JSON event to your HTTPS endpoint when a test is created, updated or deleted, and when a test suite finishes a run. Signatures follow Standard Webhooks, so each delivery carries webhook-id, webhook-timestamp and webhook-signature headers.

When to Use This Skill

• How do I receive Openlayer webhooks?
• How do I verify an Openlayer webhook signature (webhook-signature)?
• Where do I find my Openlayer webhook signing secret (whsec_...)?
• How do I handle test.created, test.updated, test.deleted or tests.result.updated events?
• How do I alert when an Openlayer test suite run has failing tests?
• Why is my Openlayer webhook verification failing, or failing only on retries and replays?

Verification (core)

Signed content is {webhook-id}.{webhook-timestamp}.{raw body}, HMAC-SHA256, base64 output. The key is the base64-decoded part of the whsec_... secret after the prefix. Compare against every v1, entry; ignore other versions. If webhook-id is absent, read svix-id, svix-timestamp and svix-signature instead, as Hookdeck does. The examples use the standardwebhooks library (npm and PyPI):

const { Webhook } = require('standardwebhooks');

// rawBody: exact request body (Buffer or string), NOT re-serialised JSON
function verifyOpenlayer(rawBody, headers, secret = process.env.OPENLAYER_WEBHOOK_SECRET) {
  const p = headers['webhook-id'] ? 'webhook' : 'svix'; // fall back to the svix-* names
  new Webhook(secret).verify(rawBody, {
    'webhook-id': headers[`${p}-id`],
    'webhook-timestamp': headers[`${p}-timestamp`],
    'webhook-signature': headers[`${p}-signature`],
  }, { jsonParse: false }); // throws on a bad signature or stale timestamp → respond 401
}

The library always enforces a fixed 5-minute timestamp window, which is the tolerance Openlayer suggests (for example 5 minutes). Hookdeck retries and replays resend the original webhook-timestamp, so the library rejects them once they are more than 5 minutes old. If you receive through Hookdeck and need late retries to pass, use the manual verifier in references/verification.md, which can drop or widen the window. In Express, read the body with express.raw({ type: '*/*' }) so any content type verifies, and parse JSON only after the check.

For complete handlers with tests, see examples/express/, examples/nextjs/, examples/fastapi/.

Event Type and Payload

Every body is { type, timestamp, data }. The event name is the top-level type, timestamp is the ISO 8601 (UTC) time the event was generated, and the event fields sit under data (data.test for test.*, data.tests for tests.result.updated):

{
  "type": "tests.result.updated",
  "timestamp": "2026-01-21T10:30:00Z",
  "data": {
    "tests": {
      "projectId": "3fa85f64-5717-4562-b3fc-2c963f66afa6",
      "projectVersionId": "3fa85f64-5717-4562-b3fc-2c963f66afa6",
      "inferencePipelineId": null,
      "total": 6, "passing": 4, "failing": 1, "skipped": 1, "running": 0
    }
  }
}

data.test.type (for example integrity) is the test's own category, not the event type.

Common Event Types

These 4 are the documented events; the list may not be exhaustive, so ignore types you don't recognise. See references/overview.md.

Delivery Semantics

Return any 2xx; the connection timeout is 5 seconds and the response timeout 10 seconds. HTTPS only. Another status, a timeout or a redirect (not followed, so a 3xx is a failure) is a failed attempt. Openlayer makes the initial attempt plus up to 3 retries after about 5 seconds, 5 minutes and 30 minutes (with up to 10% jitter), then marks the event failed. Events and delivery records are kept for 90 days. Events can repeat: deduplicate on webhook-id. Return 400 for missing headers and 401 for a bad signature.

Environment Variables

OPENLAYER_WEBHOOK_SECRET=whsec_xxxxx   # full value shown once when the subscription is created; handlers fail closed (500) without it

Local Development

# Express / Next.js (use 8000 for FastAPI). No install or account required.
npx hookdeck-cli listen 3000 openlayer --path /webhooks/openlayer

Register the printed URL as an HTTPS webhook subscription (workspace settings > Webhooks), then trigger one of the subscribed events, for example by creating a test.

Reference Materials

• references/overview.md - Event catalog, envelope and payload fields
• references/setup.md - Create a subscription in the dashboard or API, copy the secret
• references/verification.md - Library and manual verification, gotchas, debugging