Gareth Wilson
Guide to Crusoe Webhooks: Features and Best Practices
Crusoe Cloud runs GPU infrastructure for AI training and inference. Its Notifications service watches your organisation and raises an event when something significant happens: Command Center detects a hardware fault and replaces a node, a budget threshold is crossed, or a Managed AI deployment changes. You can read those events in the console or by email, or route them to Slack and to your own webhook endpoint.
This guide covers how Crusoe webhooks are configured, what Crusoe documents and what it leaves out, how to verify each delivery, and the best practices for running a Crusoe receiver in production.
What are Crusoe webhooks?
A Crusoe webhook is a generic endpoint added to the Notifications service. Crusoe doesn't run its own delivery system for these: endpoints are configured "on the Svix dashboard", which you open from the Crusoe Cloud Console. Crusoe suggests them for PagerDuty, Opsgenie or your own automation.
Crusoe's documentation is short. It describes the setup steps, the notification categories, the request method and one example body. It says nothing about signatures, signing secrets, retries or source IPs. Because delivery runs through Svix, the verification details in this guide come from Svix's documentation, and you should confirm them against your first real delivery.
Crusoe webhook features
| Feature | Details |
|---|---|
| Configuration | Svix dashboard, opened from All Notifications > Manage Slack/Webhook in the console |
| Endpoint types | Webhook for any URL, or Slack |
| Subscription model | Per endpoint, by event type |
| Method and format | POST with a JSON body |
| Body structure | Varies by event type. No type field in the documented example |
| Delivery platform | Svix |
| Signature headers | Not documented by Crusoe. Svix sends svix-id, svix-timestamp, svix-signature |
| Scheme | Svix: HMAC-SHA256 over {svix-id}.{svix-timestamp}.{raw body}, sent as v1,<base64> |
| Secret | Not documented by Crusoe. Svix secrets use the whsec_ prefix |
| Retries | Not documented |
| Source IPs | Not documented |
Common events
Crusoe groups notifications into categories, which you review on the Notification Settings page before choosing the event types for an endpoint.
| Category | What it covers |
|---|---|
| Resource Health | Hardware issues and automated remediation on your nodes |
| User Account and Organization Management | Changes to your organization's account settings and access controls |
| Billing | Spending notifications, including budget alerts |
| Managed AI | Managed AI updates and issues, for Managed AI users only |
The Managed AI notifications page splits its events into Payment Method & Budget Alerts and Managed Intelligence, which reports Self-serve deployment status updates: deployment created, deployment scaled and deployment deleted.
Only one event type string appears in Crusoe's documentation, resource.automated_. Its example body, for a node replacement, is:
{
"status": "started",
"org_name": "Org ABC 123",
"org_id": "org-abc123",
"project_name": "ml-training",
"project_id": "proj-def456",
"cluster_name": "training-cluster-prod",
"cluster_id": "cluster-abc123",
"resource_type": "VM",
"resource_name": "worker-node-42",
"resource_id": "vm-xyz789",
"location": "us-east1-a",
"workflow_type": "Node Replacement",
"triggered_by": {
"error_name": "XID 79: GPU has fallen off the bus"
}
}
There is no event type or event ID in the body, and Crusoe says "the exact structure varies by event type". The event type shows in the Svix dashboard when you subscribe, but you can't route on it from the body alone.
Setting up Crusoe webhooks
- In the Crusoe Cloud Console, click the bell icon in the top-right corner.
- Click All Notifications, then Manage Slack/Webhook in the top-right corner. This opens the Svix dashboard on a separate page.
- Click Add Endpoint and choose Webhook as the endpoint type.
- Enter your endpoint URL.
- Select the event types to subscribe to.
- Click Create.
Once the endpoint exists, copy its signing secret from the endpoint's page in the Svix dashboard. Crusoe's documentation doesn't mention the secret, so if you can't find it, ask Crusoe support.
Console notifications are always on. Some email notifications are off by default and are enabled on the Notification Settings page, separately from webhooks.
Securing Crusoe webhooks
Svix signs each message with three headers. svix-id identifies the message, svix-timestamp is the send time in Unix seconds, and svix-signature is a space-separated list of v1,<base64> entries. To verify, base64-decode the part of your secret after whsec_, compute an HMAC-SHA256 of {svix-id}.{svix-timestamp}.{raw body}, base64-encode it, and compare it in constant time with each v1, entry.
const crypto = require('crypto');
const express = require('express');
function verifyCrusoeSignature(rawBody, headers, secret) {
const id = headers['svix-id'];
const timestamp = headers['svix-timestamp'];
const signatureHeader = headers['svix-signature'];
if (!id || !timestamp || !signatureHeader || !secret) return false;
// Optional: Crusoe states no tolerance. If you want a freshness check, e.g.
// if (Math.abs(Date.now() / 1000 - Number(timestamp)) > 300) return false;
const key = Buffer.from(secret.replace(/^whsec_/, ''), 'base64');
const signedContent = Buffer.concat([Buffer.from(`${id}.${timestamp}.`, 'utf8'), rawBody]);
const expected = crypto.createHmac('sha256', key).update(signedContent).digest();
// svix-signature is a space-separated list of "v1,<base64>" entries.
return signatureHeader.split(' ').some((entry) => {
const [version, value] = entry.split(',');
if (version !== 'v1' || !value) return false;
const received = Buffer.from(value, 'base64');
return received.length === expected.length && crypto.timingSafeEqual(received, expected);
});
}
const app = express();
app.post('/webhooks/crusoe', express.raw({ type: '*/*' }), (req, res) => {
const secret = process.env.CRUSOE_WEBHOOK_SECRET;
if (!secret) return res.sendStatus(500); // fail closed
if (!verifyCrusoeSignature(req.body, req.headers, secret)) {
return res.sendStatus(401);
}
const notification = JSON.parse(req.body.toString('utf8'));
// Deduplicate on req.headers['svix-id'], then process asynchronously.
res.sendStatus(204);
});
The same check in Python:
import base64
import hashlib
import hmac
def verify_crusoe_signature(raw_body: bytes, headers, secret: str) -> bool:
msg_id = headers.get("svix-id")
timestamp = headers.get("svix-timestamp")
signature_header = headers.get("svix-signature")
if not (msg_id and timestamp and signature_header and secret):
return False
key = base64.b64decode(secret.removeprefix("whsec_"))
signed_content = f"{msg_id}.{timestamp}.".encode() + raw_body
expected = base64.b64encode(hmac.new(key, signed_content, hashlib.sha256).digest())
for entry in signature_header.split(" "):
version, _, value = entry.partition(",")
if version == "v1" and hmac.compare_digest(value.encode(), expected):
return True
return False
The svix package (npm and PyPI) does the same with new Webhook(secret).verify(rawBody, headers). It also rejects timestamps more than five minutes from your server's clock, which is the library's default rather than anything Crusoe asks for.
To check your verifier, use the secret whsec_, svix-id msg_, svix-timestamp 1761039000 and this exact body:
{"status":"started","workflow_type":"Node Replacement","resource_id":"vm-xyz789"}
The svix-signature should be:
v1,mBoc+RWoEoQhUqXP98bbLiC+XWMm0uXEnkjoNFnyZws=
Make Crusoe webhooks production-ready. Hookdeck Event Gateway verifies the Svix signature on every Crusoe notification, durably queues each node remediation and budget alert, and lets you replay any of them into your service.
Crusoe webhook limitations and pain points
The signing scheme isn't documented by Crusoe
The Problem: You want to verify Crusoe notifications, but Crusoe's documentation names no headers, algorithm or secret format.
Why It Happens: Crusoe hands delivery to Svix and documents only the setup steps. The headers, the signed content and the whsec_ key format come from Svix's documentation. Svix also lets senders use webhook-* header names instead of svix-*.
Workarounds:
- Implement the Svix scheme as shown above.
- Log the header names of your first real delivery to confirm the prefix.
- Fail closed if the secret isn't configured.
How Hookdeck Can Help: Hookdeck's Crusoe source type implements the Svix scheme. It reads the svix-* headers and falls back to webhook-*, so either prefix verifies against your whsec_ secret. Verification is optional on the source, so leave it off if you didn't set a secret.
The body doesn't say what it is
The Problem: One endpoint receives remediation, budget and organisation notifications, and you can't tell them apart reliably.
Why It Happens: Crusoe says the structure "varies by event type", and the one documented example has no type field. Only the remediation body is documented.
Workarounds:
- Give each category its own endpoint in the Svix dashboard, so the URL tells you what arrived.
- Recognise remediation bodies by
workflow_typeandresource_id, and log anything else without failing. - Capture one real delivery per event type before writing handlers for it.
How Hookdeck Can Help: Hookdeck filters can match on body fields such as workflow_type and send each kind of notification to its own destination, while the event log keeps every body for inspection.
No retry or delivery guarantees are published
The Problem: Your receiver is down during a node failure, and you don't know whether the remediation alert will be redelivered or for how long.
Why It Happens: Crusoe documents no retry schedule, response timeout or endpoint disabling rules. Svix delivers the messages, but Crusoe doesn't say how it has configured Svix's retries.
Workarounds:
- Return a
2xxas soon as the notification is stored, and process it afterwards. - Keep email notifications on for critical resource health alerts as a second channel.
- Check the console's All Notifications history after an outage, since it lists every notification, including dismissed ones.
How Hookdeck Can Help: Hookdeck accepts each Crusoe delivery as soon as it arrives and stores it, so your downtime doesn't depend on Crusoe's retry settings. You can then retry or replay stored events into your service on your own schedule.
Best practices
Verify against the raw body
The signature covers the exact bytes Svix sent. Use express.raw(), await request.text() in a Next.js route handler, or await request.body() in FastAPI, and parse JSON only after the check passes.
Deduplicate on svix-id
Svix's documentation says svix-id "will be the same when the same webhook is being resent". With no event ID in Crusoe's bodies, store the header and skip messages you've already handled. See our guide to webhook idempotency.
Treat remediation events as status updates
The example resource.automated_ body has status: "started", which suggests later updates for the same workflow. Key your state on resource_id and workflow_type, and don't assume the started event is the last you'll hear.
Acknowledge first, process after
Store the notification, return 2xx, and page your on-call team or run your automation afterwards. See why to process webhooks asynchronously.
Subscribe to what you act on
Select only the event types each endpoint handles. Slack endpoints suit people, and webhook endpoints suit automation, so split them rather than sending everything everywhere.
Conclusion
Crusoe routes its notifications through Svix, so verification follows the Svix scheme: an HMAC-SHA256 of svix-id, svix-timestamp and the raw body, keyed with your base64-decoded whsec_ secret. Crusoe documents little beyond setup and one example body, so confirm the header names on your first delivery, deduplicate on svix-id, split categories across endpoints, and parse each body defensively.
Hookdeck Event Gateway verifies each Crusoe delivery, stores it, and retries into your service on your schedule.
Get started with Hookdeck for free and handle Crusoe webhooks reliably in minutes.
