RudderStack webhooks in production: duplicates, ordering, retries and timeouts
This guide covers the problems RudderStack's HTTP Webhook destination causes in production and how to handle each one. What it sends and how it works is on the RudderStack webhooks overview, the call types are on RudderStack webhook events, and the Basic credential check is on Verify RudderStack webhooks.
RudderStack's destination pages say little about delivery failures. The retry and ordering details here come from its general FAQ on destination failures, which also says a self-hosted server can change them in config.yaml.
At a glance
| Problem | What to do | With Event Gateway |
|---|---|---|
| Timeouts | Return a 2xx quickly and process in a background job | Accepts the request at once, then queues it for your service |
| Duplicates | Deduplicate on messageId, and map it into any custom body | Drops repeats before they reach your handler |
| Ordering | Order by timestamp, and don't let one bad event block a user | Logs every request and attempt, so you can see what arrived when |
| Retries | Return 503 for temporary problems, never a 4XX | Stores each request, so a failure on your side doesn't abort the event |
| Telling events apart | Route on type, accept arrays, keep type in any mapping | Filters and routes on the body or headers |
| Testing | Use a public URL and check Live Events | Receives on your machine through the CLI and replays past requests |
| Other limits | Use HTTPS, and plan for traffic that follows your own | Delivers at a rate you set |
RudderStack webhook timeouts and slow handlers
RudderStack doesn't publish a request timeout for the HTTP Webhook destination, so return a 2xx as soon as you've authenticated and stored the request.
Why it happens: Batches can hold up to 100 events, and processing them in the request is the easiest way to run long. Because the limit isn't documented, you can't design up to it.
Fix it in your handler:
- Authenticate the request and store the raw body.
- Return
200. - Process the events from a queue. See why to process webhooks asynchronously.
With Event Gateway: Hookdeck accepts each RudderStack request immediately, then queues it and delivers it to your service, so your handler's speed doesn't affect the request RudderStack made.
Duplicate RudderStack events
Deduplicate on messageId. A request that failed after your handler did some of its work is sent again, so the same event can reach you twice.
Why it happens: RudderStack retries a 5XX or 429, and it can't tell how far your handler got. RudderStack's SDKs also retry, and it drops repeats of a messageId it has seen within 7 days during processing, but its docs say that isn't guaranteed: an event outside the window, or a node being scaled, can get through.
Fix it in your handler:
- Use
messageIdas the idempotency key. The SDKs assign it, and it stays the same when an event is retried. See our guide to webhook idempotency. - In a batch, check each event's
messageIdon its own. - If Send the event payload as is is off, map
$.messageIdinto the body, or you'll have no key. - If you set
messageIdyourself, keep it unique. RudderStack treats two events with the same value as duplicates and drops the second.
With Event Gateway: Hookdeck deduplicates requests before they reach your handler, and a field-based rule can key on messageId when batching is off.
Out-of-order RudderStack events
RudderStack keeps events in order per user, by holding back a user's later events while an earlier one is failing. It says nothing about order across users, and events can still have happened earlier than they arrive, so order by timestamp.
Why it happens: If a user's event fails, RudderStack doesn't send that user's other events until the failed one is sent or aborted. That protects order, but one event that keeps failing with a 5XX can delay that user for up to the 3 hour retry window, and once it's aborted the later events carry on without it. Clients add their own delay: RudderStack's SDKs store events locally and retry on failure.
Fix it in your handler:
- Use the
timestampfield for ordering. RudderStack calculates it to correct for client clock skew, and it saysreceivedAtdoesn't preserve chronological order and thatoriginalTimestampandsentAtreflect the client's clock. - Don't let an event you can never process keep failing. Log it, then return a
2xx. - If you replay events through RudderStack, expect older data. It keeps the original order, and warns that a destination may overwrite newer data with what it replays.
With Event Gateway: Hookdeck logs every request and attempt, so you can see the order they arrived in. Because it accepts requests at once, a failure on your side doesn't hold back that user's events in RudderStack. Keep ordering by timestamp in your service.
RudderStack webhook retries and aborted events
RudderStack retries 5XX and 429 responses for a window of 3 hours, with exponential backoff and at least 3 attempts. Other 4XX responses are aborted with no retries.
Why it happens: RudderStack treats a 4XX as a problem that retrying won't solve. That's reasonable for a bad payload, but it also covers your endpoint returning 404 during a bad deploy, or 401 while a new password reaches one side and not the other. Those events are aborted on the first attempt.
Fix it in your handler:
- Return
503for anything you expect to clear, such as a database outage or a deploy, and429if you're overloaded. - Keep
4XXfor requests you never want again, because those events won't come back. A failed credential check still needs a401, so change a password on the destination and your receiver together. - Watch the destination's Live Events tab, which shows failures and each Error Response, and alert on errors from your endpoint.
- To resend failed events, contact RudderStack Support about Event Replay. It needs a paid plan and stored event data: up to 7 days back on Growth and up to 30 on Enterprise, with no replay on Free or from Reverse ETL sources.
With Event Gateway: Hookdeck accepts and stores each request as it arrives, then delivers it to your service on a schedule you choose with retries. A 4XX from your service no longer aborts anything in RudderStack, you can replay events after an outage, and issues alert you when deliveries fail.
Telling RudderStack events apart
With the default settings, route on the body's type, and on event for a track call. The destination has no per-event subscription and adds no event-type header, so your endpoint receives every event its sources send.
Why it happens: track names are whatever your team instrumented, so RudderStack has no list of them. The settings also change the request: a mapping sends only the mapped fields, batching sends an array, and XML and form bodies aren't JSON.
Fix it in your handler:
- Accept a single object and an array, and process each event on its own so one bad event doesn't fail the rest.
- Log and acknowledge any
typeyou don't handle. The setup guide also lists Record and AudienceList messages, from warehouse and audience sources. - If you use a mapping, map
$.type,$.eventand$.messageIdinto the body, or into the path or query string. - Agree who owns the destination's settings, and review changes to them, because anyone with access can change what your endpoint receives.
With Event Gateway: Hookdeck filters match on headers and the body, so you can drop call types you don't use and route the rest when batching is off. Transformations can normalise single events and batches into one shape, and every stored request shows when the shape changed.
Testing and debugging RudderStack webhooks
Use a public URL, send a sample event, and read the result in the destination's Live Events tab.
Why it's awkward: Rudder CLI rejects localhost, bare IP addresses and ngrok.io URLs for the destination, so a local handler needs another way to be reached. What arrives also depends on your settings, so an example in the docs may not match your request. The spec pages' samples leave out the common fields.
What to do:
- Point the destination at a public URL that forwards to your machine.
hookdeck listen 3000 rudderstack --path /webhooks/rudderstackgives you one. - Send events from RudderStack's Event Playground, or your own source, and write your parser against what arrives.
- Check Live Events for delivery failures and each Error Response.
With Event Gateway: The Hookdeck CLI receives webhooks on your machine without a public server, and you can replay past requests while you work on your handler. The Hookdeck source checks Basic Auth only and has POST selected. If you change the destination's method, allow that method on the source too, and if you use Bearer Token or API Key, this source won't check the request.
Other RudderStack webhook limits
Volume follows your traffic. A campaign or a busy day sends a surge of track and page events. Batching cuts the number of requests but not the number of events, so acknowledge quickly and process from a queue. Hookdeck accepts the surge and delivers at a max delivery rate you set.
Plain HTTP is allowed. RudderStack accepts http Base URLs, and over HTTP the credentials and your users' data travel in clear text. Always use https, which needs a valid TLS certificate.
Source IPs vary by plan and region. RudderStack's FAQ lists the addresses its outbound traffic comes from. Use an allowlist as an addition to the credential check.
Keep a 4XX from losing your RudderStack events.
Event Gateway checks the Basic credentials, stores each request as it arrives and delivers it to your service with retries you control, so a failure on your side doesn't abort the event in RudderStack.