# Mailgun webhooks in production: duplicates, ordering, retries and timeouts

Agent skill: `npx skills add hookdeck/webhook-skills --skill mailgun-webhooks` ([details](https://hookdeck.com/webhooks/mailgun/skill))

This guide covers the problems Mailgun webhooks cause in production and how to handle each one. Most of it is about event webhooks, which report what happens to email you send. [Inbound Routes](#inbound-routes), which post email you receive, have their own section. What Mailgun sends and how it's signed is on the [Mailgun webhooks](/webhooks/mailgun) overview, every event type is on [Mailgun webhook events](/webhooks/mailgun/events), and signature verification code is on [Verify Mailgun webhooks](/webhooks/mailgun/verify).

## At a glance

| Problem | What to do | With Event Gateway |
| --- | --- | --- |
| [Timeouts](#timeouts) | Return exactly 200 fast and queue the work; Mailgun publishes no limit | Accepts the POST at once, then queues it for your service |
| [Duplicates](#duplicates) | Deduplicate on `event-data.id`, keeping keys for about a day | Can drop repeats by field before they reach your handler |
| [Ordering](#ordering) | Compare `timestamp` and never move a message back a state | Logs every event and attempt, so you can see arrival order when debugging |
| [Retries](#retries) | Stay available, and reconcile `delivered` from the Logs API | Accepts POSTs while your service is down and replays them |
| [Choosing events](#choosing-events) | Subscribe to what you process; match `failed` and read `severity` | Filters and routes events by their contents |
| [Testing](#testing) | Use a temporary bin, then real email through a tunnel | Forwards to localhost and replays past events |
| [Other limits](#other-limits) | Plan for regions, URL limits, truncation and redaction | Gives Mailgun one stable URL and fans out to many destinations |
| [Inbound Routes](#inbound-routes) | Handle form fields or headers, and retries of their own | Accepts the POST while your service is down |

## Mailgun webhook timeouts and slow handlers {#timeouts}

Return a 200 as fast as you can, then process the event in a background job. Mailgun's docs don't publish a response timeout or say how they treat a slow response, so don't design around a number.

**Why it happens:** Mailgun acts on the status code you return. A 200 is a success and a 406 is a rejection, and neither is retried. Mailgun retries any other code, so a handler that returns a 201 or a 204 gets the same event again even though it worked. A handler that does slow work before responding can also fail for reasons unrelated to the event, which leads to more retries.

**Fix it in your handler:**

1. Verify the signature and return a 200 straight away. Return nothing but a 200 for success.
2. Push the event onto a queue (SQS, RabbitMQ or Redis) and process it in a worker. See [why to process webhooks asynchronously](/webhooks/guides/why-implement-asynchronous-processing-webhooks).
3. Return a 406 only when you never want that POST again, such as a failed signature check. Mailgun's Go SDK example does this. Other codes, 4xx included, bring the event back.

**With Event Gateway:** Hookdeck accepts each Mailgun POST immediately, then queues it and delivers it to your service at a rate it can handle, with its own retries. A slow handler doesn't fail a Mailgun delivery.

## Duplicate Mailgun webhooks {#duplicates}

A Mailgun retry can repeat an event you already processed, for instance when your handler did the work and then returned something other than a 200. Deduplicate on `event-data.id`.

**Why it happens:** Mailgun retries whenever it doesn't see a 200 or a 406. Its docs don't say whether a retry carries the same `signature.token`, so don't use the token as your only idempotency key. Mailgun does avoid one duplicate for you: the same URL configured for the same event at both domain and account level gets that event once. Different URLs each get their own copy, and a parent account's webhook and a subaccount's webhook both fire.

**Fix it in your handler:**

1. Track processed `event-data.id` values, using a unique constraint or a fast store with an expiry. See [how to implement idempotency](/webhooks/guides/implement-webhook-idempotency).
2. Expire keys after about a day. Mailgun says the ID is unique within a day, so a key kept much longer could match a different event, and retries finish within about 8 hours.
3. Don't use `message.headers.message-id` as the key. It identifies the message, which has many events.

**With Event Gateway:** Hookdeck's [deduplication](/docs/deduplication) can use specific fields as its key, so you can drop repeats on `event-data.id` before they reach your endpoint.

## Out-of-order Mailgun webhooks {#ordering}

Mailgun's docs make no promise about delivery order. Write handlers that accept events in any order, and don't assume an `opened` arrives after its `delivered`.

**Why it happens:** A failed POST is retried minutes or hours later while newer events keep arriving. `delivered` isn't retried, so it can go missing altogether while later events for that message arrive. Mailgun also says its events are generated by physical hosts and follow different routes to its event storage, so they don't always appear in the order they occurred. For more on the general problem, see [Webhook Ordering: Why It's Hard and How to Handle It](/webhooks/guides/webhook-ordering-why-its-hard-and-how-to-handle-it).

**Fix it in your handler:**

1. Group events by `message.headers.message-id` and compare `timestamp`, which says when the event was generated, in Unix epoch seconds with a fractional part in Mailgun's examples.
2. Rank your states so a late `accepted` can't overwrite `delivered`, instead of storing whichever event came last.
3. Don't treat a temporary failure as final. `failed` with `severity` set to `temporary` means Mailgun will try again, so a `delivered` can follow it. `delivery-status.attempt-no` says which attempt it was.

**With Event Gateway:** Hookdeck logs every event and delivery attempt, so when state looks wrong you can see the order events actually arrived in and replay them.

## Mailgun webhook retries and lost events {#retries}

Mailgun retries a failed POST for about 8 hours, then stops, and it never retries the delivery notification. Keep the endpoint available and reconcile what you miss from the Logs API.

**Why it happens:** Webhook Retries lists intervals of 5, 10 and 15 minutes, then 1, 2 and 4 hours, and Mailgun's docs describe no way to change them. The page on receiving messages by HTTP lists a different set for the same behaviour (10, 15 and 30 minutes, then 1, 2 and 4 hours), so don't build logic on the exact timing. Both exclude the delivery notification, so a `delivered` event that fails during an outage isn't sent again. Mailgun's docs describe no way to resend a webhook yourself, no dead letter queue and no delivery log for webhooks.

**Fix it in your handler:**

1. Run webhook endpoints on high-availability infrastructure, and alert on your own error rate and on gaps in expected traffic.
2. Store each raw POST as it arrives, so you know what you've processed.
3. After an outage, reconcile against the [Logs API](https://documentation.mailgun.com/docs/mailgun/api-reference/send/mailgun/logs). Mailgun retains event data for 30 days, though access varies by plan, and its older Events API is deprecated. Query a range that ends well before now, because new events can still be on their way to storage. Mailgun's polling guidance uses half an hour as its example.
4. See [dead letter queues](/webhooks/guides/dead-letter-queues-webhook-reliability) for the failures on your side.

**With Event Gateway:** Hookdeck captures every Mailgun POST while your services are down, so Mailgun's deliveries succeed, including the `delivered` notifications it wouldn't send twice. Failed events can be [retried](/docs/retries) automatically, manually or in bulk once you recover.

## Choosing which Mailgun events to listen to {#choosing-events}

Subscribe only to the event types you process, then read `event-data.event`. The names you subscribe with and the names you receive aren't quite the same.

**Why it matters:** Webhooks are configured per event type, at domain or account level. There are 8 types: `accepted`, `delivered`, `temporary_fail`, `permanent_fail`, `opened`, `clicked`, `unsubscribed` and `complained`. The full list, with sample payloads, is on [Mailgun webhook events](/webhooks/mailgun/events). `rejected`, `stored` and the mailing list events appear in Mailgun's logs but can't be subscribed to as webhooks.

**Fix it in your handler:**

1. Read `event-data.event`, not a top-level `event`. Mailgun's Webhook Payloads page shows only the object inside `event-data`.
2. Match `failed`. `temporary_fail` and `permanent_fail` both arrive as `failed`, so read `event-data.severity` to tell them apart.
3. Remove addresses on `permanent_fail`, not `temporary_fail`. Read `delivery-status.bounce-type` and `reason` if you need the detail.
4. Turn on what the events depend on. `opened` and `clicked` need tracking on and the CNAME record pointing to `mailgun.org`, `unsubscribed` needs unsubscribe tracking, and `complained` depends on feedback loops.

**With Event Gateway:** Hookdeck [filters](/docs/filters) match on request contents, so you can route on `event-data.event` and `event-data.severity`, and one source can feed several [connections](/docs/connections).

## Testing and debugging Mailgun webhooks locally {#testing}

Point a webhook at a temporary bin to see what arrives, then send real email through a tunnel, because Mailgun's published examples don't show the whole body.

**Why it matters:** Mailgun needs a URL it can reach, so it can't post to `localhost`. Its Webhook Payloads examples also leave out the `signature` and `event-data` wrapper, so fixtures copied straight from that page don't match a real delivery.

**How to test it:**

1. Create a temporary URL at `bin.mailgun.net`, which Mailgun recommends for this, and set it as the webhook URL to read what Mailgun sends.
2. Send real email and follow a message from `accepted` to `delivered` to `opened` and `clicked`. Then try an address that bounces. Allow up to 10 minutes for an account-level webhook change to apply.
3. Take your endpoint offline for a while to watch the retries, and return a 406 to confirm they stop.
4. Keep fixtures of the whole body for your tests. [Mailgun webhook events](/webhooks/mailgun/events) has sample payloads to start from.

**With Event Gateway:** The [Hookdeck CLI](/docs/cli) receives webhooks on your machine without a public server, and you can replay past events with one click.

## Other Mailgun webhook limits {#other-limits}

Plan for certificates, regions, URL limits and fields that vary before they reach production.

**HTTPS certificates.** An HTTPS endpoint needs a certificate signed by a trusted CA. A self-signed certificate won't work.

**Regions.** US and EU are isolated, so create webhooks in each region you send from.

**URL limits.** Each event type can have up to 3 URLs.

**Custom variables.** `v:` variables you set when sending appear under `user-variables`, and anything over 4 KB is truncated in events and webhooks. Store large data elsewhere and send a reference.

**Fields that vary.** Mailgun says its examples aren't exhaustive and warns against hardcoding fields, because new ones can appear. Some also change shape: `user-variables` is an empty array in its examples, and `storage` is an object on most examples but an empty array on `unsubscribed`.

**Personal data.** Optional PII redaction replaces values such as `recipient` with a placeholder like `[REDACTED]`, at account or domain level. `message-id` stays intact, so correlate on that if you turn it on. Redaction is inherited by subaccounts.

**Legacy webhooks.** Mailgun's older legacy webhooks post form fields with `event` at the top level, so nothing above about `event-data` applies to them. Mailgun [made their API read only on April 15, 2023](https://www.mailgun.com/blog/product/same-api-new-tricks-get-event-notifications-just-in-time-with-webhooks/) and says they all need migrating, so move any you still have to the JSON webhooks.

**Alerts.** Mailgun's sending alerts can also go to a webhook. That's a separate feature with its own signing key and `X-Sign` header, and it isn't what these pages describe.

**With Event Gateway:** Hookdeck gives Mailgun one stable URL, so you don't use up its 3 URLs, and sends each event to as many destinations as you need.

## Mailgun Inbound Routes {#inbound-routes}

Inbound Routes are a separate surface. A route's `forward()` or `store(notify=...)` action posts email received at your domain, so there's no `event-data` or event type, and the sections above on events don't apply. Return a 200 quickly, and check which format your URL gets.

**How it differs:** For `forward()`, the content type depends on your URL and the message. A URL ending in `json` gets `application/json`, one ending in `mime` gets `application/x-www-form-urlencoded` with a raw `body-mime` field, and any other URL gets `application/x-www-form-urlencoded`, or `multipart/form-data` when the email has attachments. `store(notify=...)` posts form-encoded, or multipart with attachments. A POST carries fields such as `sender`, `recipient`, `subject`, `body-plain` and `stripped-text`.

**Set it up safely:**

1. Verify every POST with the same Webhook Signing Key. `timestamp`, `token` and `signature` are top-level form fields here, not a `signature` object.
2. Handle JSON routes differently. A URL ending in `json` gets `X-Mailgun-Signature` and `X-Mailgun-Timestamp` headers instead, and the whole request body takes the place of `token` in the HMAC.
3. Return a 200 for success and a 406 to reject a message. The same rules apply as for event webhooks, and Mailgun retries any other code for about 8 hours.

**With Event Gateway:** Hookdeck [accepts multipart requests](/docs/sources#multipart-requests) and accepts each POST while your service is down, so an outage doesn't use up Mailgun's retries. See [building AI email triage with Mailgun and Claude](/webhooks/platforms/building-ai-email-triage-with-mailgun-and-claude) for a worked example.