Guide to Estuary Webhooks: Features and Best Practices

Estuary webhooks tell the rest of your stack that fresh data has landed. Estuary moves data in real time from databases and SaaS sources into warehouses and other destinations, and a materialization trigger lets you call an HTTP endpoint every time it commits new data, for example to start a dbt run, refresh a dashboard, post to Slack or kick off your own pipeline step.

This guide covers how materialization triggers are configured, what their requests contain, how to authenticate them with a secret header, how retries and at-least-once delivery work, and the practices that stop a webhook from holding up your data.

What are Estuary webhooks?

Two Estuary features send HTTP requests to your endpoint:

• Materialization triggers are configured on a materialization. When the materialization commits a transaction, Estuary renders each trigger's payload template and sends the result to the trigger's URL. You define the URL, the method, the headers and the JSON body.
• The HTTP Webhook materialization connector sends collection data itself to an endpoint, with custom headers and a relative path per binding.

This guide concentrates on triggers, which Estuary documents in detail. The connector's documentation gives its configuration but not its method, body format or retry behaviour, so capture a request before you build on it.

A trigger fires once per committed transaction, not once per document. One transaction can contain documents from several bindings, and the collection_names variable lists the collections that contributed. Estuary adds no event type of its own: the body is whatever your template renders.

Estuary doesn't sign trigger requests. Its headers are sent as configured, and Estuary encrypts their values at rest, which makes a header the natural place for a secret your receiver checks.

Estuary webhook features

Feature	Details
Configuration	triggers.config on the materialization spec
Fires	Once per committed transaction, or at most once per interval when one is set
Methods	POST (default), PUT or PATCH
Body	A Handlebars payloadTemplate that renders JSON
Headers	headers, encrypted at rest with SOPS
Authentication	A secret header you add. No signature or timestamp
Delivery	At least once. Triggers are re-delivered after a restart
Timeout	timeout per attempt, 30 seconds by default
Retries	5xx, 408 and 429 with exponential backoff (1s, 2s, 4s, capped at 30s), up to maxAttempts (3 by default, including the first)
Other 4xx	Fail immediately, with no retry
On failure	The transaction fails and the materialization task restarts
Source IPs	Not published

What the request contains

The body is your template. Estuary provides these variables:

Variable	Contains
{{materialization_name}}	The full materialization name, such as acmeCo/example/database-views
{{collection_names}}	The collections that contributed documents to the transaction. Iterate with {{#each}}
{{connector_image}}	The materialization connector's image
{{flow_published_at_min}} and {{flow_published_at_max}}	The earliest and latest document publish timestamps in the transaction (RFC 3339)
{{run_id}}	The transaction's wall-clock start time (RFC 3339 with milliseconds)
{{headers.Name}}	The value of the header called Name, so secrets can appear in the body

Estuary's own example template renders a body like this:

{
  "materialization": "acmeCo/example/database-views",
  "collections": ["acmeCo/example/collection"],
  "publishedAtMin": "2024-01-15T08:30:00Z",
  "publishedAtMax": "2024-01-15T08:31:00Z",
  "runId": "2024-01-15T08:30:00.000Z"
}

Templates run in strict mode, so referencing an undefined variable is an error. Values aren't HTML-escaped, so the output is raw JSON, and it's up to you to keep it valid.

Inspect Estuary webhooks as they arrive. Point a trigger at a Hookdeck Inspect URL to see the exact body your template renders and the headers that arrive, with no account or setup required.

Setting up Estuary webhooks

Pick a header name and generate a long random value:

openssl rand -hex 32

Then add a trigger to the materialization spec:

materializations:
  acmeCo/example/database-views:
    # endpoint and bindings as before
    triggers:
      config:
        - url: "https://example.com/webhooks/estuary"
          method: POST
          headers:
            X-Estuary-Secret: "<your random value>"
          payloadTemplate: |
            {
              "materialization": "{{materialization_name}}",
              "collections": [{{#each collection_names}}"{{this}}"{{#unless @last}}, {{/unless}}{{/each}}],
              "runId": "{{run_id}}"
            }
          timeout: 30s
          maxAttempts: 5

Add interval under triggers (for example interval: 30m) if you only need to hear about new data periodically. A burst of transactions then collapses into at most one delivery per interval.

For the HTTP Webhook connector, set the secret header under Headers (/headers/customHeaders) as a name and value pair.

Plan the secret before you publish. Estuary's SOPS integrity check covers url, method and headers, so after the first publication you can't change any of them without re-entering every secret header value. payloadTemplate, timeout, maxAttempts and interval can be changed freely.

With Hookdeck. Estuary triggers are not signed. Add a secret header to the materialization trigger (or the HTTP Webhook connector), then create a Hookdeck source with the Estuary type and enter the same header name in API Key Header and the exact value in API Key. Hookdeck finds the header case-insensitively and compares the whole value, so for Authorization: Bearer abc123 enter Bearer abc123. Use the source URL as the trigger's url. The source pre-selects POST but doesn't manage the method, so if a trigger uses PUT or PATCH, allow that method on the source as well.

Securing Estuary webhooks

Read the header you configured, compare its whole value with the one you stored in constant time, and only then parse the body. Accept every method your triggers use.

const crypto = require('crypto');
const express = require('express');

const HEADER = (process.env.ESTUARY_WEBHOOK_HEADER || 'X-Estuary-Secret').toLowerCase();
const SECRET = process.env.ESTUARY_WEBHOOK_SECRET; // the exact, whole header value

function safeEqual(received, expected) {
  const a = Buffer.from(received, 'utf8');
  const b = Buffer.from(expected, 'utf8');
  // timingSafeEqual throws on unequal lengths, so check first.
  return a.length === b.length && crypto.timingSafeEqual(a, b);
}

function handleEstuary(req, res) {
  const received = req.get(HEADER);
  if (!SECRET || typeof received !== 'string' || !safeEqual(received, SECRET)) {
    return res.sendStatus(401);
  }

  let payload;
  try {
    payload = JSON.parse(req.body.toString('utf8'));
  } catch {
    return res.sendStatus(400);
  }

  // Deduplicate on the run ID your template sends, queue the work, then respond.
  res.sendStatus(200);
}

const app = express();
const route = '/webhooks/estuary';
const raw = express.raw({ type: '*/*' });
app.post(route, raw, handleEstuary);
app.put(route, raw, handleEstuary);
app.patch(route, raw, handleEstuary);

The same check in Python:

import hmac
import os

HEADER = os.environ.get("ESTUARY_WEBHOOK_HEADER", "X-Estuary-Secret")
SECRET = os.environ.get("ESTUARY_WEBHOOK_SECRET", "")


def verify_estuary_request(headers) -> bool:
    if not SECRET:
        return False  # fail closed when unconfigured
    received = headers.get(HEADER)  # Starlette and Flask headers are case-insensitive
    if not received:
        return False
    # Compare bytes: compare_digest raises TypeError on a non-ASCII str.
    return hmac.compare_digest(received.encode("utf-8"), SECRET.encode("utf-8"))

Be careful with the 401. Estuary doesn't retry a 4xx: it fails the trigger, which fails the transaction and restarts the materialization. A receiver that rejects a misconfigured header stops your data flowing, not just the notification.

A static header authenticates the sender, not the body. It can't detect a modified payload, and with no signed timestamp a captured request can be replayed and still pass. Use HTTPS and a long random value. Avoid copying the secret into the body with {{headers.Name}} unless the receiving API requires it, since bodies are logged more often than headers.

Make Estuary webhooks production-ready. Hookdeck Event Gateway checks your secret header, answers Estuary straight away so the transaction can complete, and delivers each trigger to your services with retries you control.

Estuary webhook limitations and pain points

A failing endpoint stalls the materialization

The Problem: Your downstream service has an outage, and data stops arriving in your warehouse too.

Why It Happens: Estuary treats a failed trigger as a failed transaction. Once maxAttempts is used up, or on any non-retryable 4xx, the materialization task restarts, and Estuary warns that "an unreachable or misconfigured webhook endpoint will block data flow for the entire materialization until the issue is resolved".

Workarounds:

• Raise maxAttempts so transient failures don't stop the task.
• Make sure the receiver answers with a 2xx as soon as the request is authenticated, and does the work afterwards.

How Hookdeck Can Help: Hookdeck answers each trigger with a 200 as soon as it's stored, so your materialization keeps committing even while the service behind Hookdeck is down. Hookdeck then delivers to that service with retries on your schedule, and you can replay anything that failed.

Short retries

The Problem: A receiver that refuses connections for more than a few seconds exhausts the retries.

Why It Happens: Retries back off from one second, double each time and cap at 30 seconds, and the default is only three attempts in total, so the default schedule spends about three seconds waiting between attempts.

Workarounds:

• Set maxAttempts to match how long your receiver can realistically be unavailable.
• Use timeout to give slow endpoints longer per attempt, rather than relying on retries.

How Hookdeck Can Help: Hookdeck's own retries to your service can run for much longer than Estuary's, and Issues alert you when deliveries are failing.

At-least-once delivery

The Problem: Your dbt job, or your own pipeline step, runs twice for the same transaction.

Why It Happens: Trigger parameters are persisted with the transaction commit, and if the materialization restarts before delivery completes, the triggers are fired again on recovery.

Workarounds:

• Put {{run_id}} in every template and deduplicate on it, as Estuary recommends.
• Make the downstream action idempotent, for example by checking whether a run for that run_id already exists.

How Hookdeck Can Help: Hookdeck's deduplication can drop repeated deliveries with the same runId before they reach your service.

Changing the header means re-entering secrets

The Problem: Rotating a header value, or adding a second header, takes more work than editing one line.

Why It Happens: The SOPS integrity check covers url, method and headers, so changing any of them after publication requires re-entering all secret header values.

Workarounds:

• Decide the header name and URL before the first publication.
• During rotation, accept both the old and new value in your receiver until the new spec is published.

How Hookdeck Can Help: Pointing triggers at a Hookdeck source keeps the URL Estuary holds fixed while you move or change the services behind it. Hookdeck's Estuary source compares the header you name with the exact value you enter, answers Estuary with a 200 either way, and records a request without the right header as failed verification, so it never reaches your destination.

Best practices

Always include run_id

It is the one value Estuary suggests for deduplication. Include materialization_name and collection_names too, so the receiver can route without a lookup.

Keep templates valid JSON

Templates run in strict mode and aren't escaped. Test them against a Hookdeck Inspect URL before pointing them at production.

Use interval for expensive downstream work

If the receiver starts something heavy, such as a full dbt build, set an interval so a busy materialization doesn't trigger it on every transaction.

Return 2xx unless you really want the task to stop

Every non-2xx response eventually fails the transaction. Reserve error responses for cases where stopping the materialization is the right outcome. See why to process webhooks asynchronously.

Make handlers idempotent

Re-delivery after a restart is normal behaviour. See our guide to webhook idempotency.

Conclusion

Estuary materialization triggers send your own Handlebars template, once per committed transaction, with at-least-once delivery and no signature. Add a secret header and compare it whole, include run_id and deduplicate on it, accept the methods your triggers use, and answer with a 2xx quickly, because a trigger that fails takes the whole materialization down with it.

Hookdeck Event Gateway sits between Estuary and your services, checks the header, answers at once so transactions keep committing, and holds every trigger until your services have processed it.

Get started with Hookdeck for free and handle Estuary webhooks reliably in minutes.