RudderStack webhooks in production: duplicates, ordering, retries and timeouts

Published · Updated

This guide covers the problems RudderStack's HTTP Webhook destination causes in production and how to handle each one. What it sends and how it works is on the RudderStack webhooks overview, the call types are on RudderStack webhook events, and the Basic credential check is on Verify RudderStack webhooks.

RudderStack's destination pages say little about delivery failures. The retry and ordering details here come from its general FAQ on destination failures, which also says a self-hosted server can change them in config.yaml.

At a glance

ProblemWhat to doWith Event Gateway
TimeoutsReturn a 2xx quickly and process in a background jobAccepts the request at once, then queues it for your service
DuplicatesDeduplicate on messageId, and map it into any custom bodyDrops repeats before they reach your handler
OrderingOrder by timestamp, and don't let one bad event block a userLogs every request and attempt, so you can see what arrived when
RetriesReturn 503 for temporary problems, never a 4XXStores each request, so a failure on your side doesn't abort the event
Telling events apartRoute on type, accept arrays, keep type in any mappingFilters and routes on the body or headers
TestingUse a public URL and check Live EventsReceives on your machine through the CLI and replays past requests
Other limitsUse HTTPS, and plan for traffic that follows your ownDelivers at a rate you set

RudderStack webhook timeouts and slow handlers

RudderStack doesn't publish a request timeout for the HTTP Webhook destination, so return a 2xx as soon as you've authenticated and stored the request.

Why it happens: Batches can hold up to 100 events, and processing them in the request is the easiest way to run long. Because the limit isn't documented, you can't design up to it.

Fix it in your handler:

  1. Authenticate the request and store the raw body.
  2. Return 200.
  3. Process the events from a queue. See why to process webhooks asynchronously.

With Event Gateway: Hookdeck accepts each RudderStack request immediately, then queues it and delivers it to your service, so your handler's speed doesn't affect the request RudderStack made.

Duplicate RudderStack events

Deduplicate on messageId. A request that failed after your handler did some of its work is sent again, so the same event can reach you twice.

Why it happens: RudderStack retries a 5XX or 429, and it can't tell how far your handler got. RudderStack's SDKs also retry, and it drops repeats of a messageId it has seen within 7 days during processing, but its docs say that isn't guaranteed: an event outside the window, or a node being scaled, can get through.

Fix it in your handler:

  1. Use messageId as the idempotency key. The SDKs assign it, and it stays the same when an event is retried. See our guide to webhook idempotency.
  2. In a batch, check each event's messageId on its own.
  3. If Send the event payload as is is off, map $.messageId into the body, or you'll have no key.
  4. If you set messageId yourself, keep it unique. RudderStack treats two events with the same value as duplicates and drops the second.

With Event Gateway: Hookdeck deduplicates requests before they reach your handler, and a field-based rule can key on messageId when batching is off.

Out-of-order RudderStack events

RudderStack keeps events in order per user, by holding back a user's later events while an earlier one is failing. It says nothing about order across users, and events can still have happened earlier than they arrive, so order by timestamp.

Why it happens: If a user's event fails, RudderStack doesn't send that user's other events until the failed one is sent or aborted. That protects order, but one event that keeps failing with a 5XX can delay that user for up to the 3 hour retry window, and once it's aborted the later events carry on without it. Clients add their own delay: RudderStack's SDKs store events locally and retry on failure.

Fix it in your handler:

  1. Use the timestamp field for ordering. RudderStack calculates it to correct for client clock skew, and it says receivedAt doesn't preserve chronological order and that originalTimestamp and sentAt reflect the client's clock.
  2. Don't let an event you can never process keep failing. Log it, then return a 2xx.
  3. If you replay events through RudderStack, expect older data. It keeps the original order, and warns that a destination may overwrite newer data with what it replays.

With Event Gateway: Hookdeck logs every request and attempt, so you can see the order they arrived in. Because it accepts requests at once, a failure on your side doesn't hold back that user's events in RudderStack. Keep ordering by timestamp in your service.

RudderStack webhook retries and aborted events

RudderStack retries 5XX and 429 responses for a window of 3 hours, with exponential backoff and at least 3 attempts. Other 4XX responses are aborted with no retries.

Why it happens: RudderStack treats a 4XX as a problem that retrying won't solve. That's reasonable for a bad payload, but it also covers your endpoint returning 404 during a bad deploy, or 401 while a new password reaches one side and not the other. Those events are aborted on the first attempt.

Fix it in your handler:

  1. Return 503 for anything you expect to clear, such as a database outage or a deploy, and 429 if you're overloaded.
  2. Keep 4XX for requests you never want again, because those events won't come back. A failed credential check still needs a 401, so change a password on the destination and your receiver together.
  3. Watch the destination's Live Events tab, which shows failures and each Error Response, and alert on errors from your endpoint.
  4. To resend failed events, contact RudderStack Support about Event Replay. It needs a paid plan and stored event data: up to 7 days back on Growth and up to 30 on Enterprise, with no replay on Free or from Reverse ETL sources.

With Event Gateway: Hookdeck accepts and stores each request as it arrives, then delivers it to your service on a schedule you choose with retries. A 4XX from your service no longer aborts anything in RudderStack, you can replay events after an outage, and issues alert you when deliveries fail.

Telling RudderStack events apart

With the default settings, route on the body's type, and on event for a track call. The destination has no per-event subscription and adds no event-type header, so your endpoint receives every event its sources send.

Why it happens: track names are whatever your team instrumented, so RudderStack has no list of them. The settings also change the request: a mapping sends only the mapped fields, batching sends an array, and XML and form bodies aren't JSON.

Fix it in your handler:

  1. Accept a single object and an array, and process each event on its own so one bad event doesn't fail the rest.
  2. Log and acknowledge any type you don't handle. The setup guide also lists Record and AudienceList messages, from warehouse and audience sources.
  3. If you use a mapping, map $.type, $.event and $.messageId into the body, or into the path or query string.
  4. Agree who owns the destination's settings, and review changes to them, because anyone with access can change what your endpoint receives.

With Event Gateway: Hookdeck filters match on headers and the body, so you can drop call types you don't use and route the rest when batching is off. Transformations can normalise single events and batches into one shape, and every stored request shows when the shape changed.

Testing and debugging RudderStack webhooks

Use a public URL, send a sample event, and read the result in the destination's Live Events tab.

Why it's awkward: Rudder CLI rejects localhost, bare IP addresses and ngrok.io URLs for the destination, so a local handler needs another way to be reached. What arrives also depends on your settings, so an example in the docs may not match your request. The spec pages' samples leave out the common fields.

What to do:

  1. Point the destination at a public URL that forwards to your machine. hookdeck listen 3000 rudderstack --path /webhooks/rudderstack gives you one.
  2. Send events from RudderStack's Event Playground, or your own source, and write your parser against what arrives.
  3. Check Live Events for delivery failures and each Error Response.

With Event Gateway: The Hookdeck CLI receives webhooks on your machine without a public server, and you can replay past requests while you work on your handler. The Hookdeck source checks Basic Auth only and has POST selected. If you change the destination's method, allow that method on the source too, and if you use Bearer Token or API Key, this source won't check the request.

Other RudderStack webhook limits

Volume follows your traffic. A campaign or a busy day sends a surge of track and page events. Batching cuts the number of requests but not the number of events, so acknowledge quickly and process from a queue. Hookdeck accepts the surge and delivers at a max delivery rate you set.

Plain HTTP is allowed. RudderStack accepts http Base URLs, and over HTTP the credentials and your users' data travel in clear text. Always use https, which needs a valid TLS certificate.

Source IPs vary by plan and region. RudderStack's FAQ lists the addresses its outbound traffic comes from. Use an allowlist as an addition to the credential check.

Keep a 4XX from losing your RudderStack events.

Event Gateway checks the Basic credentials, stores each request as it arrives and delivers it to your service with retries you control, so a failure on your side doesn't abort the event in RudderStack.

Not affiliated with RudderStack. Facts are compiled from public sources and linked inline.