Gareth Wilson Gareth Wilson

Guide to Firecrawl Webhooks: Features and Best Practices

Published

Firecrawl webhooks tell your application how a scraping job is going: a crawl starts, each page is scraped, the batch finishes, an extraction fails, an agent takes an action, or a monitor check finds changed pages. If you use Firecrawl, the web scraping and crawling API, webhooks let you process results as they arrive instead of polling the job status endpoint.

This guide covers how Firecrawl webhooks are attached to jobs and signed, the lifecycle events and their payloads, and the practices that keep a Firecrawl receiver reliable when a crawl sends hundreds of page events.

What are Firecrawl webhooks?

Firecrawl has no standing webhook subscription. Instead, you add a webhook object to each job request: a crawl, batch scrape, extract, agent job or monitor. It names the URL, optional custom headers, optional metadata to echo back, and the events you want. Firecrawl then sends JSON POST requests to that URL as the job moves through its lifecycle.

Every request is signed with one secret per Firecrawl account. The X-Firecrawl-Signature header holds sha256= followed by a hex HMAC-SHA256 of the raw body.

Firecrawl webhook features

FeatureDetails
ConfigurationA webhook object on each job request (url, headers, metadata, events)
Supported jobsCrawl, batch scrape, extract, agent, monitor
Event filterShort names such as started, page, completed, failed. Default: all
Signature headerX-Firecrawl-Signature: sha256=<hex>
SchemeHMAC-SHA256, hex
Signed contentThe raw request body only
SecretOne per account, in the Advanced tab of account settings, regenerable there
Event typetype in the body, such as crawl.page
Response window2xx within 10 seconds
Retries3: 1 minute, 5 minutes and 15 minutes after the failure
After the last retryThe webhook is marked as failed

Firecrawl does not send a timestamp and publishes no source IPs.

Common events

Firecrawl documents 16 event types.

JobEvents
Crawlcrawl.started, crawl.page, crawl.completed
Batch scrapebatch_scrape.started, batch_scrape.page, batch_scrape.completed
Extractextract.started, extract.completed, extract.failed
Agentagent.started, agent.action, agent.completed, agent.failed, agent.cancelled
Monitormonitor.page, monitor.check.completed

The events filter on a request uses the short name, such as page or completed, scoped to the job type. The delivered type is always the full name, such as crawl.page.

Every event shares the same structure. Here is Firecrawl's extract.failed example:

{
  "success": false,
  "type": "extract.failed",
  "id": "550e8400-e29b-41d4-a716-446655440000",
  "webhookId": "f1e2d3c4-0000-0000-0000-000000000010",
  "data": [],
  "error": "Failed to extract data: timeout exceeded",
  "metadata": {}
}
FieldMeaning
successWhether the operation succeeded
typeThe event type
idThe job id, the same for every event in the job
webhookIdA unique id for this webhook delivery. Sent with crawl, batch scrape, extract and monitor events, not agent events
dataEvent-specific data: scraped pages, agent actions, monitor results
metadataThe metadata you set on the request
errorThe error message when success is false

crawl.page and batch_scrape.page carry the scraped content for each page in data, so they are the largest and most frequent events. agent.action reports each tool the agent runs, with creditsUsed. monitor.page arrives as each monitored page finishes, before the whole check is reconciled, and monitor.check.completed then summarises the check with counts of same, changed, new, removed and errored pages.

Setting up Firecrawl webhooks

  1. Open the Advanced tab of your Firecrawl account settings and copy the webhook secret. Store it server-side, for example as FIRECRAWL_WEBHOOK_SECRET.
  2. Add a webhook object to the job request:
curl -X POST https://api.firecrawl.dev/v2/crawl \
  -H 'Content-Type: application/json' \
  -H "Authorization: Bearer $FIRECRAWL_API_KEY" \
  -d '{
    "url": "https://docs.firecrawl.dev",
    "limit": 100,
    "webhook": {
      "url": "https://example.com/webhooks/firecrawl",
      "metadata": { "crawl_purpose": "docs-index" },
      "events": ["started", "page", "completed"]
    }
  }'

The same object works on /v2/batch/scrape, extract, agent and monitor requests. Use metadata to carry your own ids, such as a tenant or task id, so the receiver can tell which request an event belongs to without a lookup.

Securing Firecrawl webhooks

Split X-Firecrawl-Signature on the first =, check that the algorithm is sha256, compute an HMAC-SHA256 of the raw body with your account secret, hex-encode it, and compare in constant time.

const crypto = require('crypto');
const express = require('express');

function verifyFirecrawlSignature(rawBody, signatureHeader, secret) {
  if (!signatureHeader || !secret) return false;
  const i = signatureHeader.indexOf('=');
  if (i < 0 || signatureHeader.slice(0, i) !== 'sha256') return false;

  const expected = Buffer.from(
    crypto.createHmac('sha256', secret).update(rawBody).digest('hex'),
    'utf8'
  );
  const received = Buffer.from(signatureHeader.slice(i + 1).trim().toLowerCase(), 'utf8');
  return received.length === expected.length && crypto.timingSafeEqual(received, expected);
}

const app = express();

app.post('/webhooks/firecrawl', express.raw({ type: '*/*', limit: '10mb' }), (req, res) => {
  const secret = process.env.FIRECRAWL_WEBHOOK_SECRET;
  if (!secret) return res.sendStatus(500); // fail closed

  if (!verifyFirecrawlSignature(req.body, req.get('x-firecrawl-signature'), secret)) {
    return res.sendStatus(401);
  }

  const event = JSON.parse(req.body.toString('utf8'));
  // Route on event.type; store page data and process it asynchronously.
  res.sendStatus(200);
});

The same check in Python:

import hashlib
import hmac


def verify_firecrawl_signature(raw_body: bytes, signature_header, secret: str) -> bool:
    if not signature_header or not secret:
        return False
    algorithm, _, received = signature_header.partition("=")
    if algorithm != "sha256" or not received:
        return False
    expected = hmac.new(secret.encode("utf-8"), raw_body, hashlib.sha256).hexdigest()
    return hmac.compare_digest(
        received.strip().lower().encode("utf-8"), expected.encode("utf-8")
    )


# In FastAPI: raw_body = await request.body()
# verify_firecrawl_signature(raw_body, request.headers.get("x-firecrawl-signature"),
#                            os.environ["FIRECRAWL_WEBHOOK_SECRET"])

Page events can be large. Raise your body parser's size limit, as with the limit option above, or large pages are rejected before your code runs.

Make Firecrawl webhooks production-ready. Hookdeck Event Gateway verifies the X-Firecrawl-Signature header, answers Firecrawl with an immediate 200 OK, and durably queues every page and job event before your service sees it.

Firecrawl webhook limitations and pain points

Page events arrive in bursts

The Problem: A large crawl floods your endpoint, slow handlers time out, and pages are retried or lost.

Why It Happens: A crawl sends one crawl.page event per page, each with the scraped content, as each page is scraped. Firecrawl waits only 10 seconds per request.

Workarounds:

  • Store each page and return 200 immediately, then process from a queue.
  • Subscribe only to started and completed if you plan to fetch results from the API at the end.

How Hookdeck Can Help: Hookdeck absorbs the burst, acknowledges each request immediately, and delivers to your service at a rate you set, so a 1,000-page crawl doesn't overwhelm a small worker.

Three retries in about 20 minutes

The Problem: A brief outage loses events for good.

Why It Happens: Firecrawl retries a failed delivery three times, at 1, 5 and 15 minutes after each failure, and then marks it failed.

Workarounds:

  • Keep the receiver highly available.
  • After an incident, fetch job results from the Firecrawl API rather than relying on the missing events.

How Hookdeck Can Help: Hookdeck stores every event it receives and retries delivery to your service for as long as you configure, with replay for anything that still fails.

One secret for the whole account

The Problem: Every environment and service shares one signing secret, and regenerating it affects all of them at once.

Why It Happens: Firecrawl signs every webhook with a single account-level secret, and documents no overlap period when you regenerate it.

Workarounds:

  • Keep the secret in one shared secret store, so a regeneration is one update.
  • Use metadata to separate environments and tenants, since the secret can't.

How Hookdeck Can Help: With Hookdeck in front, the secret lives on the source. Update it once and every service behind it keeps receiving verified events.

id is not a delivery id

The Problem: Deduplicating on id drops real events.

Why It Happens: id is the job id, shared by every event in the job. webhookId is described as a unique id for the delivery, but it is not sent with agent events, and Firecrawl does not say whether a retry keeps the same value.

Workarounds:

  • Deduplicate on webhookId where it exists, and check how it behaves on a retried delivery before you rely on it alone.
  • Where it is missing, build a key from id, type and identifying fields in data, such as the page URL.

How Hookdeck Can Help: Hookdeck's deduplication can use whichever body fields you choose, such as webhookId.

Best practices

Verify against the raw body

Use express.raw() in Express, await request.text() in a Next.js route handler and await request.body() in FastAPI. Firecrawl's own Express sample does the same.

Use metadata to correlate

Firecrawl echoes the request's metadata in every event. Put your own task id there, and the receiver never has to map Firecrawl job ids back to your records.

Check success before reading data

Failure events set success to false and add error. A partially failed monitor check also reports success: false. Check it before assuming data holds results.

Treat completed as the end of a job

Use crawl.completed or batch_scrape.completed to finalise a job, rather than guessing from the page events you have received. A page whose delivery is being retried can still arrive after the job is complete.

Acknowledge fast and process asynchronously

Firecrawl allows 10 seconds. Verify, store, return 200, and process afterwards. See why to process webhooks asynchronously and our guide to webhook idempotency.

Conclusion

Firecrawl signs every webhook with X-Firecrawl-Signature: sha256=<hex>, an HMAC-SHA256 of the raw body under one account-wide secret. Verify against the raw bytes, raise body size limits for page events, deduplicate on webhookId rather than the job id, use metadata to correlate jobs, and acknowledge within 10 seconds, because three retries leave little room for downtime.

Hookdeck Event Gateway sits in front of your receiver, checks every Firecrawl signature, smooths out crawl bursts, and holds each event until your service has processed it.

Get started with Hookdeck for free and handle Firecrawl webhooks reliably in minutes.

Gareth Wilson

Gareth Wilson

Product Marketing

Multi-time founding marketer, Gareth is PMM at Hookdeck and author of the newsletter, Community Inc.