# RudderStack webhooks in production: duplicates, ordering, retries and timeouts

Agent skill: `npx skills add hookdeck/webhook-skills --skill rudderstack-webhooks` ([details](https://hookdeck.com/webhooks/rudderstack/skill))

This guide covers the problems RudderStack's HTTP Webhook destination causes in production and how to handle each one. What it sends and how it works is on the [RudderStack webhooks](/webhooks/rudderstack) overview, the call types are on [RudderStack webhook events](/webhooks/rudderstack/events), and the Basic credential check is on [Verify RudderStack webhooks](/webhooks/rudderstack/verify).

RudderStack's destination pages say little about delivery failures. The retry and ordering details here come from its general FAQ on destination failures, which also says a self-hosted server can change them in `config.yaml`.

## At a glance

| Problem | What to do | With Event Gateway |
| --- | --- | --- |
| [Timeouts](#timeouts) | Return a `2xx` quickly and process in a background job | Accepts the request at once, then queues it for your service |
| [Duplicates](#duplicates) | Deduplicate on `messageId`, and map it into any custom body | Drops repeats before they reach your handler |
| [Ordering](#ordering) | Order by `timestamp`, and don't let one bad event block a user | Logs every request and attempt, so you can see what arrived when |
| [Retries](#retries) | Return `503` for temporary problems, never a `4XX` | Stores each request, so a failure on your side doesn't abort the event |
| [Telling events apart](#choosing-events) | Route on `type`, accept arrays, keep `type` in any mapping | Filters and routes on the body or headers |
| [Testing](#testing) | Use a public URL and check Live Events | Receives on your machine through the CLI and replays past requests |
| [Other limits](#other-limits) | Use HTTPS, and plan for traffic that follows your own | Delivers at a rate you set |

## RudderStack webhook timeouts and slow handlers {#timeouts}

RudderStack doesn't publish a request timeout for the HTTP Webhook destination, so return a `2xx` as soon as you've authenticated and stored the request.

**Why it happens:** Batches can hold up to 100 events, and processing them in the request is the easiest way to run long. Because the limit isn't documented, you can't design up to it.

**Fix it in your handler:**

1. Authenticate the request and store the raw body.
2. Return `200`.
3. Process the events from a queue. See [why to process webhooks asynchronously](/webhooks/guides/why-implement-asynchronous-processing-webhooks).

**With Event Gateway:** Hookdeck accepts each RudderStack request immediately, then queues it and delivers it to your service, so your handler's speed doesn't affect the request RudderStack made.

## Duplicate RudderStack events {#duplicates}

Deduplicate on `messageId`. A request that failed after your handler did some of its work is sent again, so the same event can reach you twice.

**Why it happens:** RudderStack retries a `5XX` or `429`, and it can't tell how far your handler got. RudderStack's SDKs also retry, and it drops repeats of a `messageId` it has seen within 7 days during processing, but its docs say that isn't guaranteed: an event outside the window, or a node being scaled, can get through.

**Fix it in your handler:**

1. Use `messageId` as the idempotency key. The SDKs assign it, and it stays the same when an event is retried. See our [guide to webhook idempotency](/webhooks/guides/implement-webhook-idempotency).
2. In a batch, check each event's `messageId` on its own.
3. If Send the event payload as is is off, map `$.messageId` into the body, or you'll have no key.
4. If you set `messageId` yourself, keep it unique. RudderStack treats two events with the same value as duplicates and drops the second.

**With Event Gateway:** Hookdeck [deduplicates](/docs/deduplication) requests before they reach your handler, and a field-based rule can key on `messageId` when batching is off.

## Out-of-order RudderStack events {#ordering}

RudderStack keeps events in order per user, by holding back a user's later events while an earlier one is failing. It says nothing about order across users, and events can still have happened earlier than they arrive, so order by `timestamp`.

**Why it happens:** If a user's event fails, RudderStack doesn't send that user's other events until the failed one is sent or aborted. That protects order, but one event that keeps failing with a `5XX` can delay that user for up to the 3 hour retry window, and once it's aborted the later events carry on without it. Clients add their own delay: RudderStack's SDKs store events locally and retry on failure.

**Fix it in your handler:**

1. Use the `timestamp` field for ordering. RudderStack calculates it to correct for client clock skew, and it says `receivedAt` doesn't preserve chronological order and that `originalTimestamp` and `sentAt` reflect the client's clock.
2. Don't let an event you can never process keep failing. Log it, then return a `2xx`.
3. If you replay events through RudderStack, expect older data. It keeps the original order, and warns that a destination may overwrite newer data with what it replays.

**With Event Gateway:** Hookdeck logs every request and attempt, so you can see the order they arrived in. Because it accepts requests at once, a failure on your side doesn't hold back that user's events in RudderStack. Keep ordering by `timestamp` in your service.

## RudderStack webhook retries and aborted events {#retries}

RudderStack retries `5XX` and `429` responses for a window of 3 hours, with exponential backoff and at least 3 attempts. Other `4XX` responses are aborted with no retries.

**Why it happens:** RudderStack treats a `4XX` as a problem that retrying won't solve. That's reasonable for a bad payload, but it also covers your endpoint returning `404` during a bad deploy, or `401` while a new password reaches one side and not the other. Those events are aborted on the first attempt.

**Fix it in your handler:**

1. Return `503` for anything you expect to clear, such as a database outage or a deploy, and `429` if you're overloaded.
2. Keep `4XX` for requests you never want again, because those events won't come back. A failed credential check still needs a `401`, so change a password on the destination and your receiver together.
3. Watch the destination's Live Events tab, which shows failures and each Error Response, and alert on errors from your endpoint.
4. To resend failed events, contact RudderStack Support about Event Replay. It needs a paid plan and stored event data: up to 7 days back on Growth and up to 30 on Enterprise, with no replay on Free or from Reverse ETL sources.

**With Event Gateway:** Hookdeck accepts and stores each request as it arrives, then delivers it to your service on a schedule you choose with [retries](/docs/retries). A `4XX` from your service no longer aborts anything in RudderStack, you can replay events after an outage, and [issues](/docs/issues) alert you when deliveries fail.

## Telling RudderStack events apart {#choosing-events}

With the default settings, route on the body's `type`, and on `event` for a `track` call. The destination has no per-event subscription and adds no event-type header, so your endpoint receives every event its sources send.

**Why it happens:** `track` names are whatever your team instrumented, so RudderStack has no list of them. The settings also change the request: a mapping sends only the mapped fields, batching sends an array, and XML and form bodies aren't JSON.

**Fix it in your handler:**

1. Accept a single object and an array, and process each event on its own so one bad event doesn't fail the rest.
2. Log and acknowledge any `type` you don't handle. The setup guide also lists Record and AudienceList messages, from warehouse and audience sources.
3. If you use a mapping, map `$.type`, `$.event` and `$.messageId` into the body, or into the path or query string.
4. Agree who owns the destination's settings, and review changes to them, because anyone with access can change what your endpoint receives.

**With Event Gateway:** Hookdeck [filters](/docs/filters) match on headers and the body, so you can drop call types you don't use and route the rest when batching is off. [Transformations](/docs/transformations) can normalise single events and batches into one shape, and every stored request shows when the shape changed.

## Testing and debugging RudderStack webhooks {#testing}

Use a public URL, send a sample event, and read the result in the destination's Live Events tab.

**Why it's awkward:** Rudder CLI rejects `localhost`, bare IP addresses and `ngrok.io` URLs for the destination, so a local handler needs another way to be reached. What arrives also depends on your settings, so an example in the docs may not match your request. The spec pages' samples leave out the common fields.

**What to do:**

1. Point the destination at a public URL that forwards to your machine. `hookdeck listen 3000 rudderstack --path /webhooks/rudderstack` gives you one.
2. Send events from RudderStack's Event Playground, or your own source, and write your parser against what arrives.
3. Check Live Events for delivery failures and each Error Response.

**With Event Gateway:** The [Hookdeck CLI](/docs/cli) receives webhooks on your machine without a public server, and you can replay past requests while you work on your handler. The Hookdeck source checks Basic Auth only and has `POST` selected. If you change the destination's method, allow that method on the source too, and if you use Bearer Token or API Key, this source won't check the request.

## Other RudderStack webhook limits {#other-limits}

**Volume follows your traffic.** A campaign or a busy day sends a surge of `track` and `page` events. Batching cuts the number of requests but not the number of events, so acknowledge quickly and process from a queue. Hookdeck accepts the surge and delivers at a [max delivery rate](/docs/destinations#set-a-max-delivery-rate) you set.

**Plain HTTP is allowed.** RudderStack accepts `http` Base URLs, and over HTTP the credentials and your users' data travel in clear text. Always use `https`, which needs a valid TLS certificate.

**Source IPs vary by plan and region.** RudderStack's [FAQ](https://www.rudderstack.com/docs/get-started/introduction/faq/#which-rudderstack-ips-should-i-allowlist) lists the addresses its outbound traffic comes from. Use an allowlist as an addition to the credential check.