# Guide to Estuary Webhooks: Features and Best Practices Estuary webhooks tell the rest of your stack that fresh data has landed. Estuary moves data in real time from databases and SaaS sources into warehouses and other destinations, and a materialization trigger lets you call an HTTP endpoint every time it commits new data, for example to start a dbt run, refresh a dashboard, post to Slack or kick off your own pipeline step. This guide covers how materialization triggers are configured, what their requests contain, how to authenticate them with a secret header, how retries and at-least-once delivery work, and the practices that stop a webhook from holding up your data. ## What are Estuary webhooks? Two Estuary features send HTTP requests to your endpoint: * **Materialization triggers** are configured on a materialization. When the materialization commits a transaction, Estuary renders each trigger's payload template and sends the result to the trigger's URL. You define the URL, the method, the headers and the JSON body. * **The HTTP Webhook materialization connector** sends collection data itself to an endpoint, with custom headers and a relative path per binding. This guide concentrates on triggers, which Estuary documents in detail. The connector's documentation gives its configuration but not its method, body format or retry behaviour, so capture a request before you build on it. A trigger fires once per committed transaction, not once per document. One transaction can contain documents from several bindings, and the `collection_names` variable lists the collections that contributed. Estuary adds no event type of its own: the body is whatever your template renders. Estuary doesn't sign trigger requests. Its `headers` are sent as configured, and Estuary encrypts their values at rest, which makes a header the natural place for a secret your receiver checks. ## Estuary webhook features | Feature | Details | | --- | --- | | Configuration | `triggers.config` on the materialization spec | | Fires | Once per committed transaction, or at most once per `interval` when one is set | | Methods | `POST` (default), `PUT` or `PATCH` | | Body | A Handlebars `payloadTemplate` that renders JSON | | Headers | `headers`, encrypted at rest with SOPS | | Authentication | A secret header you add. No signature or timestamp | | Delivery | At least once. Triggers are re-delivered after a restart | | Timeout | `timeout` per attempt, 30 seconds by default | | Retries | `5xx`, `408` and `429` with exponential backoff (1s, 2s, 4s, capped at 30s), up to `maxAttempts` (3 by default, including the first) | | Other `4xx` | Fail immediately, with no retry | | On failure | The transaction fails and the materialization task restarts | | Source IPs | Not published | ## What the request contains The body is your template. Estuary provides these variables: | Variable | Contains | | --- | --- | | `{{materialization_name}}` | The full materialization name, such as `acmeCo/example/database-views` | | `{{collection_names}}` | The collections that contributed documents to the transaction. Iterate with `{{#each}}` | | `{{connector_image}}` | The materialization connector's image | | `{{flow_published_at_min}}` and `{{flow_published_at_max}}` | The earliest and latest document publish timestamps in the transaction (RFC 3339) | | `{{run_id}}` | The transaction's wall-clock start time (RFC 3339 with milliseconds) | | `{{headers.Name}}` | The value of the header called `Name`, so secrets can appear in the body | Estuary's own example template renders a body like this: ```json { "materialization": "acmeCo/example/database-views", "collections": ["acmeCo/example/collection"], "publishedAtMin": "2024-01-15T08:30:00Z", "publishedAtMax": "2024-01-15T08:31:00Z", "runId": "2024-01-15T08:30:00.000Z" } ``` Templates run in strict mode, so referencing an undefined variable is an error. Values aren't HTML-escaped, so the output is raw JSON, and it's up to you to keep it valid. > **Inspect Estuary webhooks as they arrive.** Point a trigger at a [Hookdeck Inspect](https://inspect.hookdeck.com) URL to see the exact body your template renders and the headers that arrive, with no account or setup required. ## Setting up Estuary webhooks Pick a header name and generate a long random value: ```bash openssl rand -hex 32 ``` Then add a trigger to the materialization spec: ```yaml materializations: acmeCo/example/database-views: # endpoint and bindings as before triggers: config: - url: "https://example.com/webhooks/estuary" method: POST headers: X-Estuary-Secret: "" payloadTemplate: | { "materialization": "{{materialization_name}}", "collections": [{{#each collection_names}}"{{this}}"{{#unless @last}}, {{/unless}}{{/each}}], "runId": "{{run_id}}" } timeout: 30s maxAttempts: 5 ``` Add `interval` under `triggers` (for example `interval: 30m`) if you only need to hear about new data periodically. A burst of transactions then collapses into at most one delivery per interval. For the HTTP Webhook connector, set the secret header under **Headers** (`/headers/customHeaders`) as a name and value pair. Plan the secret before you publish. Estuary's SOPS integrity check covers `url`, `method` and `headers`, so after the first publication you can't change any of them without re-entering every secret header value. `payloadTemplate`, `timeout`, `maxAttempts` and `interval` can be changed freely. **With Hookdeck.** Estuary triggers are not signed. Add a secret header to the materialization trigger (or the HTTP Webhook connector), then create a Hookdeck source with the **Estuary** type and enter the same header name in **API Key Header** and the exact value in **API Key**. Hookdeck finds the header case-insensitively and compares the whole value, so for `Authorization: Bearer abc123` enter `Bearer abc123`. Use the source URL as the trigger's `url`. The source pre-selects `POST` but doesn't manage the method, so if a trigger uses `PUT` or `PATCH`, allow that method on the source as well. ## Securing Estuary webhooks Read the header you configured, compare its whole value with the one you stored in constant time, and only then parse the body. Accept every method your triggers use. ```javascript const crypto = require('crypto'); const express = require('express'); const HEADER = (process.env.ESTUARY_WEBHOOK_HEADER || 'X-Estuary-Secret').toLowerCase(); const SECRET = process.env.ESTUARY_WEBHOOK_SECRET; // the exact, whole header value function safeEqual(received, expected) { const a = Buffer.from(received, 'utf8'); const b = Buffer.from(expected, 'utf8'); // timingSafeEqual throws on unequal lengths, so check first. return a.length === b.length && crypto.timingSafeEqual(a, b); } function handleEstuary(req, res) { const received = req.get(HEADER); if (!SECRET || typeof received !== 'string' || !safeEqual(received, SECRET)) { return res.sendStatus(401); } let payload; try { payload = JSON.parse(req.body.toString('utf8')); } catch { return res.sendStatus(400); } // Deduplicate on the run ID your template sends, queue the work, then respond. res.sendStatus(200); } const app = express(); const route = '/webhooks/estuary'; const raw = express.raw({ type: '*/*' }); app.post(route, raw, handleEstuary); app.put(route, raw, handleEstuary); app.patch(route, raw, handleEstuary); ``` The same check in Python: ```python import hmac import os HEADER = os.environ.get("ESTUARY_WEBHOOK_HEADER", "X-Estuary-Secret") SECRET = os.environ.get("ESTUARY_WEBHOOK_SECRET", "") def verify_estuary_request(headers) -> bool: if not SECRET: return False # fail closed when unconfigured received = headers.get(HEADER) # Starlette and Flask headers are case-insensitive if not received: return False # Compare bytes: compare_digest raises TypeError on a non-ASCII str. return hmac.compare_digest(received.encode("utf-8"), SECRET.encode("utf-8")) ``` Be careful with the `401`. Estuary doesn't retry a `4xx`: it fails the trigger, which fails the transaction and restarts the materialization. A receiver that rejects a misconfigured header stops your data flowing, not just the notification. A static header authenticates the sender, not the body. It can't detect a modified payload, and with no signed timestamp a captured request can be replayed and still pass. Use HTTPS and a long random value. Avoid copying the secret into the body with `{{headers.Name}}` unless the receiving API requires it, since bodies are logged more often than headers. > **Make Estuary webhooks production-ready.** [Hookdeck Event Gateway](/event-gateway) checks your secret header, answers Estuary straight away so the transaction can complete, and delivers each trigger to your services with retries you control. ## Estuary webhook limitations and pain points ### A failing endpoint stalls the materialization **The Problem:** Your downstream service has an outage, and data stops arriving in your warehouse too. **Why It Happens:** Estuary treats a failed trigger as a failed transaction. Once `maxAttempts` is used up, or on any non-retryable `4xx`, the materialization task restarts, and Estuary warns that "an unreachable or misconfigured webhook endpoint will block data flow for the entire materialization until the issue is resolved". **Workarounds:** * Raise `maxAttempts` so transient failures don't stop the task. * Make sure the receiver answers with a `2xx` as soon as the request is authenticated, and does the work afterwards. **How Hookdeck Can Help:** Hookdeck answers each trigger with a `200` as soon as it's stored, so your materialization keeps committing even while the service behind Hookdeck is down. Hookdeck then delivers to that service with [retries](/docs/retries) on your schedule, and you can replay anything that failed. ### Short retries **The Problem:** A receiver that refuses connections for more than a few seconds exhausts the retries. **Why It Happens:** Retries back off from one second, double each time and cap at 30 seconds, and the default is only three attempts in total, so the default schedule spends about three seconds waiting between attempts. **Workarounds:** * Set `maxAttempts` to match how long your receiver can realistically be unavailable. * Use `timeout` to give slow endpoints longer per attempt, rather than relying on retries. **How Hookdeck Can Help:** Hookdeck's own retries to your service can run for much longer than Estuary's, and [Issues](/docs/issues) alert you when deliveries are failing. ### At-least-once delivery **The Problem:** Your dbt job, or your own pipeline step, runs twice for the same transaction. **Why It Happens:** Trigger parameters are persisted with the transaction commit, and if the materialization restarts before delivery completes, the triggers are fired again on recovery. **Workarounds:** * Put `{{run_id}}` in every template and deduplicate on it, as Estuary recommends. * Make the downstream action idempotent, for example by checking whether a run for that `run_id` already exists. **How Hookdeck Can Help:** Hookdeck's [deduplication](/docs/deduplication) can drop repeated deliveries with the same `runId` before they reach your service. ### Changing the header means re-entering secrets **The Problem:** Rotating a header value, or adding a second header, takes more work than editing one line. **Why It Happens:** The SOPS integrity check covers `url`, `method` and `headers`, so changing any of them after publication requires re-entering all secret header values. **Workarounds:** * Decide the header name and URL before the first publication. * During rotation, accept both the old and new value in your receiver until the new spec is published. **How Hookdeck Can Help:** Pointing triggers at a Hookdeck source keeps the URL Estuary holds fixed while you move or change the services behind it. Hookdeck's Estuary source compares the header you name with the exact value you enter, answers Estuary with a `200` either way, and records a request without the right header as failed verification, so it never reaches your destination. ## Best practices ### Always include `run_id` It is the one value Estuary suggests for deduplication. Include `materialization_name` and `collection_names` too, so the receiver can route without a lookup. ### Keep templates valid JSON Templates run in strict mode and aren't escaped. Test them against a [Hookdeck Inspect](https://inspect.hookdeck.com) URL before pointing them at production. ### Use `interval` for expensive downstream work If the receiver starts something heavy, such as a full dbt build, set an `interval` so a busy materialization doesn't trigger it on every transaction. ### Return `2xx` unless you really want the task to stop Every non-`2xx` response eventually fails the transaction. Reserve error responses for cases where stopping the materialization is the right outcome. See [why to process webhooks asynchronously](/webhooks/guides/why-implement-asynchronous-processing-webhooks). ### Make handlers idempotent Re-delivery after a restart is normal behaviour. See our [guide to webhook idempotency](/webhooks/guides/implement-webhook-idempotency). ## Conclusion Estuary materialization triggers send your own Handlebars template, once per committed transaction, with at-least-once delivery and no signature. Add a secret header and compare it whole, include `run_id` and deduplicate on it, accept the methods your triggers use, and answer with a `2xx` quickly, because a trigger that fails takes the whole materialization down with it. [Hookdeck Event Gateway](https://hookdeck.com) sits between Estuary and your services, checks the header, answers at once so transactions keep committing, and holds every trigger until your services have processed it. [Get started with Hookdeck](https://dashboard.hookdeck.com/signup) for free and handle Estuary webhooks reliably in minutes.