Guide to Scrapfly Webhooks: Features and Best Practices

Scrapfly webhooks notify your application when a scrape (or extraction or screenshot) job completes, delivering the full job result to your endpoint. If you're building on Scrapfly, webhooks are how you receive results asynchronously instead of holding a request open.

This guide covers how Scrapfly webhooks work, the resource types you'll handle, how to verify the X-Scrapfly-Webhook-Signature (note the uppercase hex), the two-step registration, and the best practices for production.

What are Scrapfly webhooks?

Scrapfly delivers one webhook per job on completion. Each is signed with an X-Scrapfly-Webhook-Signature header: an HMAC-SHA256 over the raw request bytes, hex-encoded in UPPERCASE. A lowercase duplicate is available in X-Scrapfly-Webhook-Signature-Lowercase. There's no timestamp header and no tolerance window, so the signature is authenticity-only. There's also no event-name field: the discriminator is the X-Scrapfly-Webhook-Resource-Type header.

Scrapfly webhook features

Common events

Scrapfly has no event catalog. Instead, the X-Scrapfly-Webhook-Resource-Type header tells you what completed:

The body is the entire job response inline (result, config, context, status, success), plus context.webhook{name, secret, consecutive_failed_count} and context.job.uuid. Other headers include X-Scrapfly-Webhook-Env, -Project, -Id, -Name, and -Job-Id.

Setting up Scrapfly webhooks

Registration is two-step, and easy to get wrong. First, create and name a webhook in the dashboard. Then, every individual scrape request must pass webhook_name=<that name>. There's no global "send all scrapes here" default, omitting webhook_name means no webhook fires at all. Store the webhook's signing secret as SCRAPFLY_WEBHOOK_SECRET.

Securing Scrapfly webhooks

Compute an HMAC-SHA256 over the raw bytes, hex-encode it, and uppercase it to match the primary header (a lowercase digest won't match X-Scrapfly-Webhook-Signature). Compare in constant time. Don't parse and re-serialize the JSON, that changes the byte sequence and breaks the signature.

const crypto = require("crypto");

const SECRET = process.env.SCRAPFLY_WEBHOOK_SECRET;

function verify(rawBody, signatureHeader) {
  if (!signatureHeader) return false;
  const expected = crypto.createHmac("sha256", SECRET).update(rawBody).digest("hex").toUpperCase();
  const received = signatureHeader.toUpperCase();
  try {
    return crypto.timingSafeEqual(Buffer.from(received, "hex"), Buffer.from(expected, "hex"));
  } catch {
    return false;
  }
}

app.post("/webhooks/scrapfly", express.raw({ type: "*/*" }), (req, res) => {
  if (!verify(req.body, req.headers["x-scrapfly-webhook-signature"])) {
    return res.sendStatus(401);
  }
  res.sendStatus(200); // acknowledge fast
  const resourceType = req.headers["x-scrapfly-webhook-resource-type"]; // scrape / extraction / screenshot
  processQueue.add({ resourceType, body: JSON.parse(req.body.toString()) }); // async
});

The same check in Python (the documented computation):

import hashlib
import hmac
import os

SECRET = os.environ["SCRAPFLY_WEBHOOK_SECRET"]


def verify(raw_body: bytes, signature_header: str) -> bool:
    expected = hmac.new(
        SECRET.encode("utf-8"), raw_body, hashlib.sha256
    ).hexdigest().upper()
    return hmac.compare_digest(expected, (signature_header or "").upper())

Scrapfly webhook limitations and pain points

The primary signature is uppercase hex

The Problem: The primary X-Scrapfly-Webhook-Signature is uppercase hex. A constant-time compare against a lowercase digest fails against it.

Why It Happens: Scrapfly uppercases the hex digest (.hexdigest().upper()).

Workarounds:

• Uppercase your computed digest (or normalize both sides), or verify against the -Lowercase companion header.

How Hookdeck Can Help: Hookdeck verifies the signature at the edge, so casing doesn't trip your app.

There's no event field

The Problem: Scrapfly has no event-name field in the body. Code that switches on a type field has nothing to switch on.

Why It Happens: The job type is carried in the X-Scrapfly-Webhook-Resource-Type header, not the body.

Workarounds:

• Dispatch on the X-Scrapfly-Webhook-Resource-Type header (scrape / extraction / screenshot).

How Hookdeck Can Help: Hookdeck's filters can route on the resource-type header, giving you type-based routing.

Two-step registration is easy to miss

The Problem: Naming a webhook in the dashboard isn't enough. If a scrape request doesn't pass webhook_name, no webhook fires, silently.

Why It Happens: Scrapfly has no global default destination; delivery is opt-in per request.

Workarounds:

• Pass webhook_name=<name> on every job you want delivered.

How Hookdeck Can Help: Hookdeck gives you a stable endpoint and full delivery visibility, so you can see immediately whether jobs are arriving.

Endpoints auto-disable after 100 failures

The Problem: After 100 consecutive failures Scrapfly auto-disables the webhook, and it must be re-enabled by hand in the dashboard. Deliveries stop until you notice.

Why It Happens: Scrapfly disables persistently failing endpoints.

Workarounds:

• Acknowledge reliably, and monitor for the disabled state.

How Hookdeck Can Help: Hookdeck absorbs delivery to your app behind a durable queue and retries, so a downstream blip doesn't march toward the 100-failure cutoff.

Best practices

Verify uppercase HMAC-SHA256 over the raw bytes

Compute the hex HMAC over the raw bytes, uppercase it, and compare against the primary header in constant time; never re-serialize the body first.

Dispatch on the resource-type header

Route on X-Scrapfly-Webhook-Resource-Type, not a body field.

Always pass webhook_name

Every scrape you want delivered must include webhook_name=<name>.

Acknowledge fast, process asynchronously

Return 200 quickly and defer work to a queue; dedupe on X-Scrapfly-Webhook-Id / context.job.uuid. See why to process webhooks asynchronously.

Conclusion

Scrapfly delivers one webhook per job, verified with an X-Scrapfly-Webhook-Signature HMAC-SHA256 over the raw bytes, hex in UPPERCASE (with a lowercase duplicate available). There's no timestamp and no event field, dispatch on X-Scrapfly-Webhook-Resource-Type. Uppercase your digest to match, never re-serialize the body, pass webhook_name on every job, and watch for the 100-failure auto-disable.

Hookdeck verifies the signature, deduplicates, and durably queues every job result at the edge, so your app only ever processes verified, unique events.

Get started with Hookdeck for free and handle Scrapfly webhooks reliably in minutes.