Microsoft Graph webhooks in production: duplicates, ordering, retries and timeouts
This guide covers the problems Microsoft Graph change notifications cause in production and how to handle each one. What Graph sends is on the Microsoft Graph webhooks overview, every change type and lifecycle event is on Microsoft Graph webhook events, and the handshake and clientState check are on Verify Microsoft Graph webhooks.
At a glance
| Problem | What to do | With Event Gateway |
|---|---|---|
| Timeouts | Queue the notification and return a 202 within 3 seconds | Accepts each delivery, then queues it and delivers at a rate you set |
| Duplicates | Loop over value, and make each item safe to process twice | Can drop identical deliveries within a window you set |
| Ordering | Treat a notification as a prompt, and fetch current state | Logs every event and attempt, so you can see arrival order when debugging |
| Retries | Return a 5xx if you can't queue, and resync with a delta query | Holds events and retries them on your schedule, and you can replay failures |
| Choosing events | Pick a resource and the changeType values it supports | Filters and routes requests on the body |
| Testing | Tunnel to localhost, and test the handshake and a batch | Answers the validation request, with sample payloads in Inspect and replay with the CLI |
| Subscription expiry | Renew before expiry, and handle lifecycle notifications | Queues and retries lifecycle notifications, so a warning survives your outage |
Microsoft Graph webhook timeouts
Return a 2xx within 3 seconds. If you can't finish the work by then, queue the notification and return a 202 Accepted.
Why it happens: Microsoft considers a notification delivered once it gets a 2xx within 3 seconds. A timeout or a non-2xx response starts retries, which get a 10-second timeout. Slow responses also change how Graph treats your endpoint, measured over a 10-minute window:
- Slow: when more than 10% of responses take longer than 3 seconds, new notifications arrive after a 10-minute delay.
- Drop: when more than 15% of responses take longer than 10 seconds, Graph drops notifications for 10 minutes, and they can't be recovered.
Graph tests each state with a small number of notifications before it lets your endpoint out.
Fix it in your handler:
- Check
clientState, write the notification to a queue and return a202. See why to process webhooks asynchronously. - Keep the request path free of Graph API calls. Fetching the changed resource belongs in the worker.
- On rich notifications, respond before you validate the tokens. Microsoft says to respond even if validation fails later, which avoids needless retries and hides the result from an attacker.
- If your endpoint can't keep up, Microsoft suggests delivering to Event Hubs or Event Grid instead.
With Event Gateway: Hookdeck accepts each delivery, then queues it and delivers to your service at a rate it can handle, with its own retries. Your handler can take as long as it needs without counting against Graph's window.
Duplicate Microsoft Graph notifications
Assume a notification can arrive more than once, and expect one POST to hold several changes.
Why it happens: Graph retries a delivery it didn't get a timely 2xx for, so a handler that finishes its work and responds late can see it again. Microsoft's docs don't mention duplicates or a delivery guarantee. The id field is a unique ID for the notification, but it's marked optional and the lifecycle template doesn't show it. Separately, Graph can batch notifications for different subscriptions into one POST.
Fix it in your handler:
- Loop over
valueand process each item independently. A handler that reads onlyvalue[0]loses the rest. - Dedupe on
idwhere it's present. Where it isn't, fall back to a key from the fields you have, and remember that Microsoft doesn't promise any of them is stable across retries. - Make every handler idempotent. Fetching the current state of
resourceand writing it is safe to repeat, which also lets two real updates share a key without harm.
def dedupe_key(item):
if item.get("id"):
return item["id"]
etag = (item.get("resourceData") or {}).get("@odata.etag", "")
return f"{item['subscriptionId']}:{item.get('changeType')}:{item.get('resource')}:{etag}"
With Event Gateway: Hookdeck can deduplicate deliveries that match within a window of up to an hour. It compares the whole payload or the fields you choose. Deduplication is best effort, so keep your handler idempotent too.
Out-of-order Microsoft Graph notifications
Microsoft's docs don't promise an order. Treat each notification as a prompt to read the resource, not as a record of the change.
Why it happens: Retries run on backoff for hours, and a slow endpoint gets new notifications after a 10-minute delay, so an earlier change can arrive after a later one. Latency also varies by resource: a message averages under a minute with a 3-minute maximum, while a OneDrive driveItem can take up to 6 hours. For more on the general problem, see Webhook Ordering: Why It's Hard and How to Handle It.
Fix it in your handler:
- Use
resourceto fetch the current state from Graph, rather than applying changes in the order they arrive. - Handle a
404on that fetch, since a resource can be deleted before you read it. - Use a delta query where you need a complete, consistent view of a resource.
With Event Gateway: Hookdeck logs every event and delivery attempt, so when state looks wrong you can see the order events actually arrived in and replay them.
Microsoft Graph webhook retries
Graph retries a failed delivery for up to 4 hours. Beyond that, or after a drop, the only way to get the data back is to resync.
Why it happens: After a non-2xx response or a timeout, Graph keeps trying for up to 4 hours with exponential backoff. Microsoft says dropped notifications can't be recovered, and its docs describe no way to resend one. An expired access token also stops delivery, though it doesn't count towards throttling: if you refresh the token within 4 hours of expiry, the notifications held back are delivered.
Graph can tell you about a gap, but not for everything. The missed lifecycle notification only covers Outlook messages, events and personal contacts.
Fix it in your handler:
- Return a
5xxif you couldn't process or queue a notification, so Graph retries it. - On a
missedlifecycle notification, return a202and then run a full resync of the resource, for example with a delta query. - After an outage on any other resource, run the same resync, because nothing will tell you what you lost.
With Event Gateway: Hookdeck accepts deliveries while your service is down, retries them on your schedule and lets you replay failures, so an outage on your side doesn't use up Graph's 4 hours. It can't help if the subscription has lapsed, since Graph sends nothing to anyone once it's gone.
Choosing Microsoft Graph resources and change types
Graph has no catalogue of named events. You choose a resource and one or more changeType values, and those two together define what you receive.
changeType and resource | You're notified when |
|---|---|
created,updated on /me/mailFolders('inbox')/messages | A message arrives in or is updated in the inbox |
updated,deleted on /users | A user is updated or permanently deleted |
created,updated on /teams/{id}/channels/{id}/messages | A Teams channel message is posted or changes |
updated on /drives/{id}/root | A driveItem in that drive changes |
Why it matters: Not every resource supports every change type. Drive root items and lists support only updated. Users and groups support updated and deleted: updated covers a user or group being created, updated or soft deleted, and deleted means permanently deleted. Graph also rejects a second subscription with the same changeType and resource, with a 409 Conflict.
Fix it in your handler:
- Start with a basic notification, which gives you an ID to fetch. It's the simpler path, because a rich notification needs an encryption certificate and a decryption step.
- Choose rich notifications only if the extra API calls are the problem. They can shorten the subscription as well: on Outlook resources, the lifetime falls from under seven days to under one day.
- Read
resourceandchangeTypetogether when you route an item.updatedon its own doesn't tell you what changed. - Add a
lifecycleNotificationUrlwhen you create the subscription, because you can't add one later. It can be the same URL asnotificationUrl, in which case your handler tells them apart by whether an item haschangeTypeorlifecycleEvent.
With Event Gateway: Hookdeck's filters match on the request body, so you can send notifications to different services. A POST can hold several items, so match with that in mind, or give lifecycle notifications their own source.
Testing Microsoft Graph webhooks
Test the validation handshake first, because a subscription doesn't exist until it passes.
Why it's hard: Graph needs a public HTTPS URL and sends the validation request while your POST /subscriptions call is still waiting. If you get the answer wrong, the subscription call fails and no notification is ever sent. The answer needs a 200, text/plain and the URL-decoded token, within 10 seconds. There is no test button: real changes to a real resource are what produce notifications.
How to test it:
- Get a public URL.
hookdeck listen 3000 microsoft-graph --path /webhooks/microsoft-graphforwards to your machine and prints a URL to use asnotificationUrl. - Create a short-lived subscription. Graph sets any expiry under 45 minutes to 45 minutes, so you don't need to manage a long one while testing.
- Change the resource, for example by sending a message to the mailbox, and wait for its usual latency.
- Replay a sample body with two items in
value, one with a wrongclientState, and check only the good one is processed. Sample bodies are on the events page.
With Event Gateway: Hookdeck answers the validation request for you, so the subscription call succeeds whether or not your handler is running. Keep the clientState check in your handler. You can browse sample payloads in Hookdeck Inspect, then forward deliveries to localhost and replay them with the Hookdeck CLI.
Subscription expiry and reauthorization
Subscriptions expire and access tokens run out on separate clocks, and either one can stop your notifications. Renew on a schedule and handle lifecycle notifications.
Why it happens: Graph deletes a subscription when it reaches its expirationDateTime, then sends nothing more. The maximum lifetime depends on the resource:
| Resource | Maximum lifetime |
|---|---|
presence | 60 minutes |
| Teams Shifts resources | 6 hours |
Teams chatMessage, channel and chat | 3 days |
Outlook message, event and contact | Under 7 days, or under 1 day with resource data |
user, group and other directory resources | Under 29 days |
OneDrive driveItem and SharePoint list | Under 30 days |
The access token behind a subscription generally expires within an hour, so it can run out first. Delivery then pauses until you act, and changes that happen during a long pause are lost, so you have to fetch them separately.
Lifecycle notifications warn you, but coverage varies: reauthorizationRequired covers all resources, subscriptionRemoved covers Outlook message, event and personal contact plus Teams chatMessage, and missed covers those Outlook resources. Teams resources require a lifecycleNotificationUrl if the expiry is more than an hour away.
Fix it in your handler:
- Store
expirationDateTime, or readsubscriptionExpirationDateTimefrom each notification, andPATCH /subscriptions/{id}with a newexpirationDateTimewell before it passes. Renewing also reauthorizes the subscription and refreshes the token. - On
reauthorizationRequired, return a202, then eitherPATCHa new expiry or callPOST /subscriptions/{id}/reauthorize. Don't send both for the same subscription within 10 minutes, which Microsoft warns can leave it in an inconsistent state. - On
subscriptionRemoved, return a202, create a new subscription, then resync what you missed. - If a
PATCHreturns404, the subscription has gone. Create a new one and resync. - Run a scheduled job that lists your subscriptions and recreates missing ones, since lifecycle notifications don't cover every case.
Microsoft also publishes the IP addresses it delivers from, so you can limit inbound connections at your firewall.
With Event Gateway: Point lifecycleNotificationUrl at its own Hookdeck source and connection, so lifecycle notifications are queued and retried separately from your change notifications. A reauthorizationRequired warning then still reaches your service after it comes back from an outage. Renewing the subscription stays with your app, because that call is made with your access token.
Keep a slow handler from getting your Graph notifications dropped.
Hookdeck Event Gateway answers Graph's validation request, accepts each notification, queues it and retries to your service on your schedule, so your handler can take as long as it needs.