# Microsoft Graph webhooks in production: duplicates, ordering, retries and timeouts

Agent skill: `npx skills add hookdeck/webhook-skills --skill microsoft-graph-webhooks` ([details](https://hookdeck.com/webhooks/microsoft-graph/skill))

This guide covers the problems Microsoft Graph change notifications cause in production and how to handle each one. What Graph sends is on the [Microsoft Graph webhooks](/webhooks/microsoft-graph) overview, every change type and lifecycle event is on [Microsoft Graph webhook events](/webhooks/microsoft-graph/events), and the handshake and `clientState` check are on [Verify Microsoft Graph webhooks](/webhooks/microsoft-graph/verify).

## At a glance

| Problem | What to do | With Event Gateway |
| --- | --- | --- |
| [Timeouts](#timeouts) | Queue the notification and return a `202` within 3 seconds | Accepts each delivery, then queues it and delivers at a rate you set |
| [Duplicates](#duplicates) | Loop over `value`, and make each item safe to process twice | Can drop identical deliveries within a window you set |
| [Ordering](#ordering) | Treat a notification as a prompt, and fetch current state | Logs every event and attempt, so you can see arrival order when debugging |
| [Retries](#retries) | Return a `5xx` if you can't queue, and resync with a delta query | Holds events and retries them on your schedule, and you can replay failures |
| [Choosing events](#choosing-events) | Pick a `resource` and the `changeType` values it supports | Filters and routes requests on the body |
| [Testing](#testing) | Tunnel to localhost, and test the handshake and a batch | Answers the validation request, with sample payloads in Inspect and replay with the CLI |
| [Subscription expiry](#other-limits) | Renew before expiry, and handle lifecycle notifications | Queues and retries lifecycle notifications, so a warning survives your outage |

## Microsoft Graph webhook timeouts {#timeouts}

Return a 2xx within 3 seconds. If you can't finish the work by then, queue the notification and return a `202 Accepted`.

**Why it happens:** Microsoft [considers a notification delivered](https://learn.microsoft.com/en-us/graph/change-notifications-delivery-webhooks#http-codes-and-retry-logic) once it gets a 2xx within 3 seconds. A timeout or a non-2xx response starts retries, which get a 10-second timeout. Slow responses also change how Graph treats your endpoint, measured over a 10-minute window:

* **Slow:** when more than 10% of responses take longer than 3 seconds, new notifications arrive after a 10-minute delay.
* **Drop:** when more than 15% of responses take longer than 10 seconds, Graph drops notifications for 10 minutes, and they can't be recovered.

Graph tests each state with a small number of notifications before it lets your endpoint out.

**Fix it in your handler:**

1. Check `clientState`, write the notification to a queue and return a `202`. See [why to process webhooks asynchronously](/webhooks/guides/why-implement-asynchronous-processing-webhooks).
2. Keep the request path free of Graph API calls. Fetching the changed resource belongs in the worker.
3. On rich notifications, respond before you validate the tokens. Microsoft says to respond [even if validation fails later](https://learn.microsoft.com/en-us/graph/change-notifications-with-resource-data#how-to-validate), which avoids needless retries and hides the result from an attacker.
4. If your endpoint can't keep up, Microsoft suggests delivering to [Event Hubs or Event Grid](https://learn.microsoft.com/en-us/graph/change-notifications-delivery-webhooks#throttling) instead.

**With Event Gateway:** Hookdeck accepts each delivery, then queues it and delivers to your service at a rate it can handle, with its own retries. Your handler can take as long as it needs without counting against Graph's window.

## Duplicate Microsoft Graph notifications {#duplicates}

Assume a notification can arrive more than once, and expect one POST to hold several changes.

**Why it happens:** Graph retries a delivery it didn't get a timely 2xx for, so a handler that finishes its work and responds late can see it again. Microsoft's docs don't mention duplicates or a delivery guarantee. The [`id` field](https://learn.microsoft.com/en-us/graph/api/resources/changenotification) is a unique ID for the notification, but it's marked optional and the lifecycle template doesn't show it. Separately, [Graph can batch](https://learn.microsoft.com/en-us/graph/change-notifications-delivery-webhooks#change-notification-example) notifications for different subscriptions into one POST.

**Fix it in your handler:**

1. Loop over `value` and process each item independently. A handler that reads only `value[0]` loses the rest.
2. Dedupe on `id` where it's present. Where it isn't, fall back to a key from the fields you have, and remember that Microsoft doesn't promise any of them is stable across retries.
3. Make every handler [idempotent](/webhooks/guides/implement-webhook-idempotency). Fetching the current state of `resource` and writing it is safe to repeat, which also lets two real updates share a key without harm.

```python
def dedupe_key(item):
    if item.get("id"):
        return item["id"]
    etag = (item.get("resourceData") or {}).get("@odata.etag", "")
    return f"{item['subscriptionId']}:{item.get('changeType')}:{item.get('resource')}:{etag}"

```

**With Event Gateway:** Hookdeck can [deduplicate](/docs/deduplication) deliveries that match within a window of up to an hour. It compares the whole payload or the fields you choose. Deduplication is best effort, so keep your handler idempotent too.

## Out-of-order Microsoft Graph notifications {#ordering}

Microsoft's docs don't promise an order. Treat each notification as a prompt to read the resource, not as a record of the change.

**Why it happens:** Retries run on backoff for hours, and a slow endpoint gets new notifications after a 10-minute delay, so an earlier change can arrive after a later one. [Latency also varies by resource](https://learn.microsoft.com/en-us/graph/change-notifications-overview#latency): a message averages under a minute with a 3-minute maximum, while a OneDrive `driveItem` can take up to 6 hours. For more on the general problem, see [Webhook Ordering: Why It's Hard and How to Handle It](/webhooks/guides/webhook-ordering-why-its-hard-and-how-to-handle-it).

**Fix it in your handler:**

1. Use `resource` to fetch the current state from Graph, rather than applying changes in the order they arrive.
2. Handle a `404` on that fetch, since a resource can be deleted before you read it.
3. Use a [delta query](https://learn.microsoft.com/en-us/graph/delta-query-overview) where you need a complete, consistent view of a resource.

**With Event Gateway:** Hookdeck logs every event and delivery attempt, so when state looks wrong you can see the order events actually arrived in and replay them.

## Microsoft Graph webhook retries {#retries}

Graph retries a failed delivery for up to 4 hours. Beyond that, or after a drop, the only way to get the data back is to resync.

**Why it happens:** After a non-2xx response or a timeout, Graph [keeps trying for up to 4 hours](https://learn.microsoft.com/en-us/graph/change-notifications-delivery-webhooks#http-codes-and-retry-logic) with exponential backoff. Microsoft says dropped notifications can't be recovered, and its docs describe no way to resend one. An expired access token also stops delivery, though it doesn't count towards throttling: if you refresh the token within 4 hours of expiry, the notifications held back are delivered.

Graph can tell you about a gap, but not for everything. The [`missed` lifecycle notification](https://learn.microsoft.com/en-us/graph/change-notifications-lifecycle-events#missed-notifications) only covers Outlook messages, events and personal contacts.

**Fix it in your handler:**

1. Return a `5xx` if you couldn't process or queue a notification, so Graph retries it.
2. On a `missed` lifecycle notification, return a `202` and then run a full resync of the resource, for example with a [delta query](https://learn.microsoft.com/en-us/graph/delta-query-overview).
3. After an outage on any other resource, run the same resync, because nothing will tell you what you lost.

**With Event Gateway:** Hookdeck accepts deliveries while your service is down, retries them on [your schedule](/docs/retries) and lets you replay failures, so an outage on your side doesn't use up Graph's 4 hours. It can't help if the subscription has lapsed, since Graph sends nothing to anyone once it's gone.

## Choosing Microsoft Graph resources and change types {#choosing-events}

Graph has no catalogue of named events. You choose a `resource` and one or more `changeType` values, and those two together define what you receive.

| `changeType` and `resource` | You're notified when |
| --- | --- |
| `created,updated` on `/me/mailFolders('inbox')/messages` | A message arrives in or is updated in the inbox |
| `updated,deleted` on `/users` | A user is updated or permanently deleted |
| `created,updated` on `/teams/{id}/channels/{id}/messages` | A Teams channel message is posted or changes |
| `updated` on `/drives/{id}/root` | A `driveItem` in that drive changes |

**Why it matters:** Not every resource supports every change type. [Drive root items and lists](https://learn.microsoft.com/en-us/graph/api/resources/subscription) support only `updated`. Users and groups support `updated` and `deleted`: `updated` covers a user or group being created, updated or soft deleted, and `deleted` means permanently deleted. Graph also rejects a second subscription with the same `changeType` and `resource`, with a `409 Conflict`.

**Fix it in your handler:**

1. Start with a basic notification, which gives you an ID to fetch. It's the simpler path, because a rich notification needs an encryption certificate and a decryption step.
2. Choose [rich notifications](https://learn.microsoft.com/en-us/graph/change-notifications-with-resource-data) only if the extra API calls are the problem. They can shorten the subscription as well: on Outlook resources, the lifetime falls from under seven days to under one day.
3. Read `resource` and `changeType` together when you route an item. `updated` on its own doesn't tell you what changed.
4. Add a `lifecycleNotificationUrl` when you create the subscription, because you can't add one later. It can be the same URL as `notificationUrl`, in which case your handler tells them apart by whether an item has `changeType` or `lifecycleEvent`.

**With Event Gateway:** Hookdeck's [filters](/docs/filters) match on the request body, so you can send notifications to different services. A POST can hold several items, so match with that in mind, or give lifecycle notifications their own source.

## Testing Microsoft Graph webhooks {#testing}

Test the validation handshake first, because a subscription doesn't exist until it passes.

**Why it's hard:** Graph needs a public HTTPS URL and sends the validation request while your `POST /subscriptions` call is still waiting. If you get the answer wrong, the subscription call fails and no notification is ever sent. The answer needs a `200`, `text/plain` and the URL-decoded token, [within 10 seconds](https://learn.microsoft.com/en-us/graph/change-notifications-delivery-webhooks#notificationurl-validation). There is no test button: real changes to a real resource are what produce notifications.

**How to test it:**

1. Get a public URL. `hookdeck listen 3000 microsoft-graph --path /webhooks/microsoft-graph` forwards to your machine and prints a URL to use as `notificationUrl`.
2. Create a short-lived subscription. Graph sets any expiry under 45 minutes to 45 minutes, so you don't need to manage a long one while testing.
3. Change the resource, for example by sending a message to the mailbox, and wait for its [usual latency](https://learn.microsoft.com/en-us/graph/change-notifications-overview#latency).
4. Replay a sample body with two items in `value`, one with a wrong `clientState`, and check only the good one is processed. Sample bodies are on the [events page](/webhooks/microsoft-graph/events).

**With Event Gateway:** Hookdeck answers the validation request for you, so the subscription call succeeds whether or not your handler is running. Keep the `clientState` check in your handler. You can browse sample payloads in [Hookdeck Inspect](https://inspect.hookdeck.com/webhooks/microsoft-graph), then forward deliveries to localhost and replay them with the [Hookdeck CLI](/docs/cli).

## Subscription expiry and reauthorization {#other-limits}

Subscriptions expire and access tokens run out on separate clocks, and either one can stop your notifications. Renew on a schedule and handle lifecycle notifications.

**Why it happens:** [Graph deletes a subscription](https://learn.microsoft.com/en-us/graph/change-notifications-delivery-webhooks#subscription-lifecycle) when it reaches its `expirationDateTime`, then sends nothing more. The [maximum lifetime](https://learn.microsoft.com/en-us/graph/change-notifications-overview#subscription-lifetime) depends on the resource:

| Resource | Maximum lifetime |
| --- | --- |
| `presence` | 60 minutes |
| Teams Shifts resources | 6 hours |
| Teams `chatMessage`, `channel` and `chat` | 3 days |
| Outlook `message`, `event` and `contact` | Under 7 days, or under 1 day with resource data |
| `user`, `group` and other directory resources | Under 29 days |
| OneDrive `driveItem` and SharePoint `list` | Under 30 days |

The access token behind a subscription generally expires within an hour, so it can run out first. Delivery then pauses until you act, and changes that happen during a long pause [are lost](https://learn.microsoft.com/en-us/graph/change-notifications-lifecycle-events#reauthorizationrequired-notifications), so you have to fetch them separately.

Lifecycle notifications warn you, but [coverage varies](https://learn.microsoft.com/en-us/graph/change-notifications-lifecycle-events#supported-resources): `reauthorizationRequired` covers all resources, `subscriptionRemoved` covers Outlook `message`, `event` and personal `contact` plus Teams `chatMessage`, and `missed` covers those Outlook resources. Teams resources require a `lifecycleNotificationUrl` if the expiry is more than an hour away.

**Fix it in your handler:**

1. Store `expirationDateTime`, or read `subscriptionExpirationDateTime` from each notification, and `PATCH /subscriptions/{id}` with a new `expirationDateTime` well before it passes. Renewing also reauthorizes the subscription and refreshes the token.
2. On `reauthorizationRequired`, return a `202`, then either `PATCH` a new expiry or call `POST /subscriptions/{id}/reauthorize`. Don't send both for the same subscription within 10 minutes, which Microsoft warns can leave it in an inconsistent state.
3. On `subscriptionRemoved`, return a `202`, create a new subscription, then resync what you missed.
4. If a `PATCH` returns `404`, the subscription has gone. Create a new one and resync.
5. Run a scheduled job that lists your subscriptions and recreates missing ones, since lifecycle notifications don't cover every case.

Microsoft also publishes the [IP addresses it delivers from](https://learn.microsoft.com/en-us/graph/change-notifications-delivery-webhooks#firewall-configuration), so you can limit inbound connections at your firewall.

**With Event Gateway:** Point `lifecycleNotificationUrl` at its own Hookdeck source and connection, so lifecycle notifications are queued and retried separately from your change notifications. A `reauthorizationRequired` warning then still reaches your service after it comes back from an outage. Renewing the subscription stays with your app, because that call is made with your access token.