Slack webhooks in production: duplicates, ordering, retries and timeouts

Published · Updated

This guide covers the problems Slack webhooks cause in production and how to handle each one. What Slack sends and how it's signed is on the Slack webhooks overview, every event type is on Slack webhook events, and signature verification code is on Verify Slack webhooks.

Slack uses the word webhook for features that run in opposite directions. The Events API sends requests to your Request URL, and most of this guide is about it. Incoming webhooks are requests you send to a channel's URL, and their limits are in Rate limits and incoming webhooks. To send another service's events into Slack, see how to send external webhooks to Slack.

At a glance

ProblemWhat to doWith Event Gateway
TimeoutsReturn a 2xx within 3 seconds; process in a background jobAccepts the delivery at once, then queues it for your service
DuplicatesRecord processed event_id values; make handlers idempotentDrops repeats before they reach your handler
OrderingDon't rely on arrival order; fetch current state when order mattersLogs every event and attempt, so you can see arrival order when debugging
RetriesStay under the 95% failure limit; turn on Delayed Events; reconcile after an outageAccepts deliveries while your service is down and retries them
Choosing eventsSubscribe to what you process; switch on event.type, not the subscription nameFilters and routes events by type
TestingUse Socket Mode or a tunnel; test the challenge and a bad signatureSample payloads in Inspect, and forwarding and replay with the CLI
Rate limits and incoming webhooksAlert on app_rate_limited; respect Retry-After; keep webhook URLs secretQueues what you send to Slack and delivers it at a rate you set

Slack webhook timeouts and slow handlers

Respond with an HTTP 2xx within 3 seconds, then do the work in a background job.

Why it happens: Slack's Events API treats a response that takes longer than three seconds as a failed attempt, and nothing in its docs lets you extend that. The retry then arrives with x-slack-retry-reason: http_timeout, and it counts against your failure limit. Handlers that write to a database or call other APIs before responding are the usual cause, and so are serverless cold starts, which use up part of the 3 seconds before your code runs.

Fix it in your handler:

  1. Verify the signature and return a 200 straight away.
  2. Push the event onto a queue and process it in a worker. See why to process webhooks asynchronously.
  3. Keep the request path free of calls to other services, including the Slack Web API.

Socket Mode doesn't remove this work. You still acknowledge each event so Slack knows whether to retry, and Slack recommends HTTP for production.

With Event Gateway: Hookdeck accepts each delivery within Slack's limit, then delivers events to your service with its own timeouts and retries. Your handler can take as long as it needs.

Duplicate Slack webhook events

Assume an event can arrive more than once, and deduplicate on event_id.

Why it happens: Slack retries any attempt it counts as failed, so a handler that finishes its work and then responds late sees the same event again. Slack describes event_id as a unique identifier for the event, globally unique across workspaces. Its docs don't state outright that a retry repeats the value, though a retry is a repeat of that event.

Fix it in your handler:

  1. Record the event_id of each event you process, in a table with a unique constraint or a store with a TTL.
  2. Make every handler idempotent: the same result however many times it runs.
  3. Don't drop a request because it has an x-slack-retry-num header. That header says Slack is retrying, not that you finished the first attempt.
async function processSlackEvent(payload) {
  const { event_id, event } = payload;

  // SET NX returns null when the key already exists
  const first = await redis.set(`processed:${event_id}`, '1', 'EX', 86400, 'NX');
  if (!first) return;

  try {
    await handleEvent(event);
  } catch (err) {
    // Let a later attempt process it
    await redis.del(`processed:${event_id}`);
    throw err;
  }
}

With Event Gateway: Hookdeck deduplicates events before they reach your endpoint, so repeats are filtered at the infrastructure level rather than in each handler.

Out-of-order Slack webhook events

Slack makes no general promise about delivery order, so don't rely on it.

Why it happens: Each event is its own HTTP request, and retries come up to 5 minutes later, or up to 24 hours later with Delayed Events, so a later event can arrive first. Slack calls the Events API a best-effort system whose delivery can be delayed during incidents. The one ordering statement in its docs is on app_uninstalled: you can receive tokens_revoked after it. For more on the general problem, see Webhook Ordering: Why It's Hard and How to Handle It.

Fix it in your handler:

  1. Compare event.event_ts before you apply an update, so an older event can't overwrite newer state. event_time at the top level is when Slack dispatched the event.
  2. When order matters, treat the event as a signal and fetch current state from the Web API, such as conversations.info for a channel.
  3. Handle app_uninstalled and tokens_revoked in either order.

With Event Gateway: Hookdeck logs every event and delivery attempt, so when state looks wrong you can see the order events actually arrived in and replay them.

Slack webhook retries and disabled subscriptions

Slack retries three times over a few minutes, and if most deliveries keep failing it turns your event subscriptions off. Keep the endpoint up, turn on Delayed Events, and reconcile after any outage.

Why it happens: The first retry is nearly immediate, the second comes after 1 minute and the third after 5 minutes. Each carries x-slack-retry-num and an x-slack-retry-reason of http_timeout, too_many_redirects, connection_failed, ssl_error, http_error or unknown_error. Slack's docs describe no way to resend an event or browse past deliveries. Delivery can also stop altogether:

  • The failure limit. If timeouts, SSL failures, more than 2 redirects or non-2xx responses make up more than 95% of delivery attempts within 60 minutes, Slack temporarily disables your event subscriptions and emails the app's creator and owner. Retries count towards this. Apps receiving fewer than 1,000 events an hour aren't automatically disabled.
  • The two-hour cutoff. By default Slack won't attempt an event more than two hours late, which matters during Slack's own incidents as well as yours.

Fix it in your handler:

  1. Turn on Delayed Events under Event Subscriptions. Slack then follows the three retries with hourly retries for 24 hours, and attempts late events regardless of the two-hour cutoff.
  2. Make sure someone reads the emails sent to your app's owner, and re-enable deliveries in your app's settings once the endpoint is fixed.
  3. Return x-slack-no-retry: 1 with a non-200 response when you don't want an event sent again. Slack's docs say only that it stops re-delivery of that event, so don't assume the response leaves your failure count alone.
  4. After an outage, reconcile what the Web API can tell you. conversations.history returns messages you may have missed.

With Event Gateway: Hookdeck acknowledges Slack's deliveries while your service is down, so your outage doesn't count against Slack's failure limit, and it retries to your service on your schedule, well past Slack's few minutes. You can inspect failed attempts and replay them once the endpoint is fixed.

Choosing which Slack events to listen to

Subscribe only to the events you process, and route on event.type. The name you subscribe to and the value you receive don't always match.

Why it matters: Your app has one Request URL for every event and every workspace, so all of it arrives at one handler. Several names on the event reference also behave differently from how they read:

  • Subscription names. message.channels, message.groups, message.im and message.mpim are names you subscribe to. Each delivery has event.type: message, and event.channel_type says where it was posted.
  • Message subtypes. Edits, deletions, joins and similar changes arrive as event.type: message with an event.subtype such as message_changed. Not all subtypes are deliverable in the Events API.
  • No inner event. url_verification and app_rate_limited have no event object, so their name is in the outer type.
  • RTM-only events. The reference also lists events for the legacy RTM API, such as hello, user_typing and presence_change, which don't reach a Request URL. Each page's Compatible APIs says which. The list on Slack webhook events leaves those out.

Fix it in your handler:

  1. Check the outer type first, then switch on event.type.
  2. Use app_mention if your app only needs messages that mention it, instead of subscribing to every message in a channel. Direct messages don't trigger it, so add message.im for those.
  3. Check for bot_id when you handle message events, so your app can tell a bot's message from a person's.
  4. Remember that scopes control delivery. An event you subscribe to doesn't arrive until the installing user has granted its OAuth scope, so adding a new scope means sending existing users through OAuth again.
  5. Slack sends one event when several installed users can see it, with one user in authorizations. Call apps.event.authorizations.list if you need all of them.

With Event Gateway: Hookdeck filters and routes events by type or any other property, so one Slack Request URL can feed several services.

Testing and debugging Slack webhooks locally

Use Socket Mode or a tunnel, and test the URL challenge and a bad signature as well as real events.

Why it happens: Slack needs a public HTTPS URL and won't save it until your endpoint answers the challenge, so your handler has to be running at that moment. A failed challenge means no events at all, with nothing to inspect. Request URLs are case-sensitive, and an endpoint that's slow to wake up can use the Retry button in your app's settings.

Fix it in your handler:

  1. For early development, use Socket Mode, which Slack recommends for working locally because it needs no public URL. Test over HTTP before you deploy, since signature checks and the challenge only happen there.
  2. Answer url_verification with HTTP 200 and the challenge value, as plain text, form-encoded or JSON.
  3. Send your endpoint a request with a bad signature and one with an old timestamp, and check that both are rejected.
  4. Generate real events in a test workspace: post a message, add a reaction, mention your app. Slack's docs describe no way to send a test event.
  5. Try the failure paths too: a handler that throws, and one that takes longer than 3 seconds.

With Event Gateway: Hookdeck Inspect shows sample Slack payloads with no account or setup. The Hookdeck CLI forwards events to your machine and replays past ones, so you can test a fix without triggering the event again in Slack.

Rate limits and incoming webhooks

The Events API cap. Event deliveries max out at 30,000 per workspace per app per 60 minutes. Above that Slack sends app_rate_limited every minute you're limited, with minute_rate_limited saying when it began. The limit applies to what Slack sends, so a busy workspace can reach it however fast your endpoint is. Slack's docs don't say whether withheld events are sent later, so alert on this event, reconcile from the Web API, and drop subscriptions you don't need.

The rest of this section is about incoming webhooks, where you're the sender. They have no signature, retries or delivery log on Slack's side, so handle failures in your own code.

Rate limits. Apps may post no more than one message per second per channel, with short bursts allowed. Over the limit, Slack returns a 429 with a Retry-After header giving the seconds to wait. Queue messages on your side and respect that header.

Failed sends. Success is a 200 with the body ok. Errors come back as strings alongside status codes such as 400, 403 and 404. Don't retry invalid_payload, action_prohibited, channel_not_found or channel_is_archived without fixing the cause. no_service means the webhook has been disabled, removed or is invalid.

The URL is the credential. It contains a secret, and anyone who has it can post to your channel. Keep it out of source control and logs, since Slack actively searches out and revokes leaked secrets, which also stops your messages.

One channel per URL. You can't override the channel, username or icon in the payload, so posting to ten channels takes ten URLs. Call chat.postMessage with a bot token if you need to choose the channel at runtime or delete messages later.

Threads. An incoming webhook can post into a thread with thread_ts, but it doesn't return the ts of what you sent. You'd get that from a message event, conversations.history or assistant.search.context.

Message limits. A message can have at most 100 attachments, and more returns too_many_attachments. Include text alongside blocks, since it's the fallback, and a payload with no text can return no_text.

With Event Gateway: Send to Hookdeck and make your Slack URL its destination. You set a maximum delivery rate, Hookdeck queues what's above it and retries failures, a transformation can turn another service's event into a Slack message, and your sources never see the Slack URL. Hookdeck can't raise the Events API cap, because Slack applies that before anything is delivered.

Keep an outage from disabling your Slack event subscriptions.

Hookdeck Event Gateway answers Slack's URL verification, accepts every delivery within Slack's 3 seconds, verifies the signature, queues events and retries to your service on your schedule.

Not affiliated with Slack. Facts are compiled from public sources and linked inline.