Microsoft Graph webhooks in production: duplicates, ordering, retries and timeouts

Published · Updated

This guide covers the problems Microsoft Graph change notifications cause in production and how to handle each one. What Graph sends is on the Microsoft Graph webhooks overview, every change type and lifecycle event is on Microsoft Graph webhook events, and the handshake and clientState check are on Verify Microsoft Graph webhooks.

At a glance

ProblemWhat to doWith Event Gateway
TimeoutsQueue the notification and return a 202 within 3 secondsAccepts each delivery, then queues it and delivers at a rate you set
DuplicatesLoop over value, and make each item safe to process twiceCan drop identical deliveries within a window you set
OrderingTreat a notification as a prompt, and fetch current stateLogs every event and attempt, so you can see arrival order when debugging
RetriesReturn a 5xx if you can't queue, and resync with a delta queryHolds events and retries them on your schedule, and you can replay failures
Choosing eventsPick a resource and the changeType values it supportsFilters and routes requests on the body
TestingTunnel to localhost, and test the handshake and a batchAnswers the validation request, with sample payloads in Inspect and replay with the CLI
Subscription expiryRenew before expiry, and handle lifecycle notificationsQueues and retries lifecycle notifications, so a warning survives your outage

Microsoft Graph webhook timeouts

Return a 2xx within 3 seconds. If you can't finish the work by then, queue the notification and return a 202 Accepted.

Why it happens: Microsoft considers a notification delivered once it gets a 2xx within 3 seconds. A timeout or a non-2xx response starts retries, which get a 10-second timeout. Slow responses also change how Graph treats your endpoint, measured over a 10-minute window:

  • Slow: when more than 10% of responses take longer than 3 seconds, new notifications arrive after a 10-minute delay.
  • Drop: when more than 15% of responses take longer than 10 seconds, Graph drops notifications for 10 minutes, and they can't be recovered.

Graph tests each state with a small number of notifications before it lets your endpoint out.

Fix it in your handler:

  1. Check clientState, write the notification to a queue and return a 202. See why to process webhooks asynchronously.
  2. Keep the request path free of Graph API calls. Fetching the changed resource belongs in the worker.
  3. On rich notifications, respond before you validate the tokens. Microsoft says to respond even if validation fails later, which avoids needless retries and hides the result from an attacker.
  4. If your endpoint can't keep up, Microsoft suggests delivering to Event Hubs or Event Grid instead.

With Event Gateway: Hookdeck accepts each delivery, then queues it and delivers to your service at a rate it can handle, with its own retries. Your handler can take as long as it needs without counting against Graph's window.

Duplicate Microsoft Graph notifications

Assume a notification can arrive more than once, and expect one POST to hold several changes.

Why it happens: Graph retries a delivery it didn't get a timely 2xx for, so a handler that finishes its work and responds late can see it again. Microsoft's docs don't mention duplicates or a delivery guarantee. The id field is a unique ID for the notification, but it's marked optional and the lifecycle template doesn't show it. Separately, Graph can batch notifications for different subscriptions into one POST.

Fix it in your handler:

  1. Loop over value and process each item independently. A handler that reads only value[0] loses the rest.
  2. Dedupe on id where it's present. Where it isn't, fall back to a key from the fields you have, and remember that Microsoft doesn't promise any of them is stable across retries.
  3. Make every handler idempotent. Fetching the current state of resource and writing it is safe to repeat, which also lets two real updates share a key without harm.
def dedupe_key(item):
    if item.get("id"):
        return item["id"]
    etag = (item.get("resourceData") or {}).get("@odata.etag", "")
    return f"{item['subscriptionId']}:{item.get('changeType')}:{item.get('resource')}:{etag}"

With Event Gateway: Hookdeck can deduplicate deliveries that match within a window of up to an hour. It compares the whole payload or the fields you choose. Deduplication is best effort, so keep your handler idempotent too.

Out-of-order Microsoft Graph notifications

Microsoft's docs don't promise an order. Treat each notification as a prompt to read the resource, not as a record of the change.

Why it happens: Retries run on backoff for hours, and a slow endpoint gets new notifications after a 10-minute delay, so an earlier change can arrive after a later one. Latency also varies by resource: a message averages under a minute with a 3-minute maximum, while a OneDrive driveItem can take up to 6 hours. For more on the general problem, see Webhook Ordering: Why It's Hard and How to Handle It.

Fix it in your handler:

  1. Use resource to fetch the current state from Graph, rather than applying changes in the order they arrive.
  2. Handle a 404 on that fetch, since a resource can be deleted before you read it.
  3. Use a delta query where you need a complete, consistent view of a resource.

With Event Gateway: Hookdeck logs every event and delivery attempt, so when state looks wrong you can see the order events actually arrived in and replay them.

Microsoft Graph webhook retries

Graph retries a failed delivery for up to 4 hours. Beyond that, or after a drop, the only way to get the data back is to resync.

Why it happens: After a non-2xx response or a timeout, Graph keeps trying for up to 4 hours with exponential backoff. Microsoft says dropped notifications can't be recovered, and its docs describe no way to resend one. An expired access token also stops delivery, though it doesn't count towards throttling: if you refresh the token within 4 hours of expiry, the notifications held back are delivered.

Graph can tell you about a gap, but not for everything. The missed lifecycle notification only covers Outlook messages, events and personal contacts.

Fix it in your handler:

  1. Return a 5xx if you couldn't process or queue a notification, so Graph retries it.
  2. On a missed lifecycle notification, return a 202 and then run a full resync of the resource, for example with a delta query.
  3. After an outage on any other resource, run the same resync, because nothing will tell you what you lost.

With Event Gateway: Hookdeck accepts deliveries while your service is down, retries them on your schedule and lets you replay failures, so an outage on your side doesn't use up Graph's 4 hours. It can't help if the subscription has lapsed, since Graph sends nothing to anyone once it's gone.

Choosing Microsoft Graph resources and change types

Graph has no catalogue of named events. You choose a resource and one or more changeType values, and those two together define what you receive.

changeType and resourceYou're notified when
created,updated on /me/mailFolders('inbox')/messagesA message arrives in or is updated in the inbox
updated,deleted on /usersA user is updated or permanently deleted
created,updated on /teams/{id}/channels/{id}/messagesA Teams channel message is posted or changes
updated on /drives/{id}/rootA driveItem in that drive changes

Why it matters: Not every resource supports every change type. Drive root items and lists support only updated. Users and groups support updated and deleted: updated covers a user or group being created, updated or soft deleted, and deleted means permanently deleted. Graph also rejects a second subscription with the same changeType and resource, with a 409 Conflict.

Fix it in your handler:

  1. Start with a basic notification, which gives you an ID to fetch. It's the simpler path, because a rich notification needs an encryption certificate and a decryption step.
  2. Choose rich notifications only if the extra API calls are the problem. They can shorten the subscription as well: on Outlook resources, the lifetime falls from under seven days to under one day.
  3. Read resource and changeType together when you route an item. updated on its own doesn't tell you what changed.
  4. Add a lifecycleNotificationUrl when you create the subscription, because you can't add one later. It can be the same URL as notificationUrl, in which case your handler tells them apart by whether an item has changeType or lifecycleEvent.

With Event Gateway: Hookdeck's filters match on the request body, so you can send notifications to different services. A POST can hold several items, so match with that in mind, or give lifecycle notifications their own source.

Testing Microsoft Graph webhooks

Test the validation handshake first, because a subscription doesn't exist until it passes.

Why it's hard: Graph needs a public HTTPS URL and sends the validation request while your POST /subscriptions call is still waiting. If you get the answer wrong, the subscription call fails and no notification is ever sent. The answer needs a 200, text/plain and the URL-decoded token, within 10 seconds. There is no test button: real changes to a real resource are what produce notifications.

How to test it:

  1. Get a public URL. hookdeck listen 3000 microsoft-graph --path /webhooks/microsoft-graph forwards to your machine and prints a URL to use as notificationUrl.
  2. Create a short-lived subscription. Graph sets any expiry under 45 minutes to 45 minutes, so you don't need to manage a long one while testing.
  3. Change the resource, for example by sending a message to the mailbox, and wait for its usual latency.
  4. Replay a sample body with two items in value, one with a wrong clientState, and check only the good one is processed. Sample bodies are on the events page.

With Event Gateway: Hookdeck answers the validation request for you, so the subscription call succeeds whether or not your handler is running. Keep the clientState check in your handler. You can browse sample payloads in Hookdeck Inspect, then forward deliveries to localhost and replay them with the Hookdeck CLI.

Subscription expiry and reauthorization

Subscriptions expire and access tokens run out on separate clocks, and either one can stop your notifications. Renew on a schedule and handle lifecycle notifications.

Why it happens: Graph deletes a subscription when it reaches its expirationDateTime, then sends nothing more. The maximum lifetime depends on the resource:

ResourceMaximum lifetime
presence60 minutes
Teams Shifts resources6 hours
Teams chatMessage, channel and chat3 days
Outlook message, event and contactUnder 7 days, or under 1 day with resource data
user, group and other directory resourcesUnder 29 days
OneDrive driveItem and SharePoint listUnder 30 days

The access token behind a subscription generally expires within an hour, so it can run out first. Delivery then pauses until you act, and changes that happen during a long pause are lost, so you have to fetch them separately.

Lifecycle notifications warn you, but coverage varies: reauthorizationRequired covers all resources, subscriptionRemoved covers Outlook message, event and personal contact plus Teams chatMessage, and missed covers those Outlook resources. Teams resources require a lifecycleNotificationUrl if the expiry is more than an hour away.

Fix it in your handler:

  1. Store expirationDateTime, or read subscriptionExpirationDateTime from each notification, and PATCH /subscriptions/{id} with a new expirationDateTime well before it passes. Renewing also reauthorizes the subscription and refreshes the token.
  2. On reauthorizationRequired, return a 202, then either PATCH a new expiry or call POST /subscriptions/{id}/reauthorize. Don't send both for the same subscription within 10 minutes, which Microsoft warns can leave it in an inconsistent state.
  3. On subscriptionRemoved, return a 202, create a new subscription, then resync what you missed.
  4. If a PATCH returns 404, the subscription has gone. Create a new one and resync.
  5. Run a scheduled job that lists your subscriptions and recreates missing ones, since lifecycle notifications don't cover every case.

Microsoft also publishes the IP addresses it delivers from, so you can limit inbound connections at your firewall.

With Event Gateway: Point lifecycleNotificationUrl at its own Hookdeck source and connection, so lifecycle notifications are queued and retried separately from your change notifications. A reauthorizationRequired warning then still reaches your service after it comes back from an outage. Renewing the subscription stays with your app, because that call is made with your access token.

Keep a slow handler from getting your Graph notifications dropped.

Hookdeck Event Gateway answers Graph's validation request, accepts each notification, queues it and retries to your service on your schedule, so your handler can take as long as it needs.

Not affiliated with Microsoft Graph. Facts are compiled from public sources and linked inline.