Grafana webhooks in production: duplicates, ordering, retries and timeouts

Published · Updated

This guide covers the problems the Webhook integration in Grafana Alerting causes in production and how to handle each one. What Grafana sends is on the Grafana webhooks overview, firing and resolved are on Grafana webhook events, and signature verification code is on Verify Grafana webhooks.

Where Grafana's docs leave something out, this guide follows its current source code, since Grafana is open source: the webhook notifier, the HTTP client and Grafana's Alertmanager fork. It doesn't cover Grafana IRM's outgoing webhooks, which are a separate feature.

At a glance

ProblemWhat to doWith Event Gateway
TimeoutsReturn a 2xx within 30 seconds; process in a background jobAccepts the notification at once, then queues it for your service
DuplicatesKey each alert on fingerprint, startsAt and statusDrops repeats before they reach your handler
OrderingRead alerts[].status; compare startsAt for each fingerprintLogs every event and attempt, so you can see arrival order when debugging
RetriesStay available; expire incidents that stop being repeatedAccepts notifications while your service is down and retries them
Choosing alertsRoute with notification policies, Group by and the timersFilters and routes on status and labels
TestingUse the Test button, then make a real rule fire and resolveForwards to localhost and replays past events
Other limitsCheck truncatedAlerts; plan for Custom Payload and header limitsReshapes bodies and delivers at a rate you set

Grafana webhook timeouts and slow handlers

Grafana gives your endpoint 30 seconds to respond, and you can't change that on a contact point. Return a 2xx first, then do the work in a background job.

Why it happens: Grafana's notifier HTTP client sets a 30-second timeout on the whole request, and the webhook integration has no setting for it. Any 2xx is a success. Anything else, including a timeout, is a failure.

Handlers usually run long because of what they call next, such as a ticketing API or an on-call service, or because of a serverless cold start. One notification can also hold many alerts, so work done per alert grows with the size of the group. A handler that finishes after Grafana has given up still did the work, and Grafana sends the group again at its next interval, so a slow handler produces duplicates too.

Fix it in your handler:

  1. Verify the signature and return 200 straight away.
  2. Push the raw body onto a queue and process it in a worker. See why to process webhooks asynchronously.
  3. Make calls to other services from the worker, with their own timeouts and retries.

With Event Gateway: Hookdeck accepts each Grafana notification immediately, so Grafana records a success and moves on. It then delivers to your service with a timeout you can configure, 60 seconds by default and up to 15 minutes per destination, with its own retries.

Duplicate Grafana webhooks

Grafana sends the same alert more than once by design, and there's no delivery ID. Key your side effects on each alert's fingerprint, startsAt and status.

Why it happens: There are four common causes:

  • The repeat interval. Grafana resends a group that hasn't changed as a reminder that its alerts are still firing, every 4 hours by default.
  • Group changes. At each group interval, 5 minutes by default, a group that has changed gets a new notification. It lists the group's current alerts, including ones you've already seen.
  • High availability. Every Grafana instance evaluates every rule by default, and the instances gossip to avoid sending twice. Grafana describes this as best effort: "Alertmanager chooses availability over consistency, which may result in occasional duplicated or out-of-order notifications." If the instances can't reach each other, expect more of them.
  • A slow success. If your handler finishes after the 30-second limit, Grafana counts a failure and sends the group again.

fingerprint alone isn't enough. It's a hash of the alert's labels, so it stays the same every time an alert with those labels fires. Keying on fingerprint and status would swallow the second incident on the same server. Adding startsAt separates them, and a repeat reminder then counts as a duplicate, which is usually what you want.

Fix it in your handler:

  1. Loop over alerts[] and build a key for each alert from fingerprint, startsAt and status.
  2. Skip any alert whose key you've already processed. See how to implement webhook idempotency.
  3. On self-hosted high availability, check your ha_peers setup and consider raising ha_peer_timeout, 15 seconds by default, if your instances are far apart.
def handle_grafana_notification(body):
    for alert in body["alerts"]:
        key = f'{alert["fingerprint"]}:{alert["startsAt"]}:{alert["status"]}'

        # Already handled this alert in this state
        if not cache.set(f"grafana_alert_{key}", True, nx=True, ex=604800):
            continue

        process(alert)

With Event Gateway: Hookdeck deduplicates events before they reach your endpoint, on the whole payload or on fields you choose, so repeated notifications are filtered at the infrastructure level. It works per notification, so keep the per-alert check if a group can change between sends.

Out-of-order Grafana webhooks

Grafana doesn't promise an order, and its high availability docs say notifications can arrive out of order. Read each alert's own status, and compare startsAt with what you've stored for that fingerprint.

Why it happens: Grafana's high availability docs say Alertmanager prefers sending a notification to keeping its instances consistent. Each group also runs on its own timers, so notifications for different groups have no order between them. For more on the general problem, see Webhook Ordering: Why It's Hard and How to Handle It.

The top-level status can mislead you as well. It's firing if any alert in the group is firing, so a firing notification can list alerts that have already resolved. Each notification describes the group as it stood when Grafana sent it.

Fix it in your handler:

  1. Act on alerts[].status, not the top-level status.
  2. Store the latest startsAt for each fingerprint, and ignore an alert older than that.
  3. Use endsAt to spot a resolved alert. It's 0001-01-01T00:00:00Z while an alert is firing.

With Event Gateway: Hookdeck logs every event and delivery attempt, so when an incident's state looks wrong you can see the order notifications actually arrived in and replay them.

Grafana webhook retries and lost notifications

Grafana doesn't retry a failed webhook straight away. It tries again when the group's next interval comes round, and during a long outage it drops resolved alerts, so stay available and don't depend on resolved arriving.

Why it happens: When a webhook request fails, the notifier tells Alertmanager not to retry, so there's one attempt, whatever the status code. Grafana's log shows notify retry canceled due to unrecoverable error after 1 attempts. Returning a 4xx rather than a 5xx makes no difference.

A failed notification isn't written to the notification log, so at the next group interval, 5 minutes by default, Grafana finds nothing recorded and sends the group again. That carries on for as long as the group has alerts, and Grafana's docs describe nothing that disables a contact point because it keeps failing. Two things can still be lost:

  • Resolved alerts during an outage. If notifications keep failing, Grafana deletes resolved alerts after 3 group intervals, 15 minutes by default. An outage longer than that can leave an incident open on your side with no resolved to close it.
  • Alerts that came and went. If a group's firing notification never succeeded and its alerts then resolve, Grafana sends nothing, because as far as it knows you never heard about them.

Grafana also has no queue you can drain and no way to resend a notification by hand.

Fix it in your handler:

  1. Keep the endpoint available, or put a queue in front of it.
  2. Watch delivery from the Contact points tab and notification history. Use a different contact point to alert on failures, since a broken one can't report itself.
  3. Treat the repeat reminder as a heartbeat. An incident with no notification for longer than your repeat interval may have ended without you hearing, so check it against Grafana's active alerts.

With Event Gateway: Hookdeck acknowledges each notification, so Grafana sees a success even while your service is down. Events wait in a queue, retry on the schedule you configure and can be replayed once your service recovers.

Choosing which Grafana alerts to send

You can't subscribe to event types, because Grafana has none. A contact point receives what notification policies route to it, so use those to control what reaches your endpoint and how often.

Why it happens: Each notification is for one group of alerts, and firing and resolved are that group's status. A policy matches alerts by label and picks the contact point. Group by decides which alerts share a notification, and by default that's by alert rule, using alertname and grafana_folder.

Fix it in your setup:

  1. Add a child policy with label matchers, so only the alerts this service handles reach its contact point.
  2. Set Group by to match how you handle alerts. Grouping by the special label ... sends each alert on its own, which also means more requests.
  3. Tune group wait, group interval and repeat interval. Their defaults are 30 seconds, 5 minutes and 4 hours.
  4. Only turn on Disable resolved message if nothing depends on resolved, because Grafana then sends no notification when a group's alerts all end.

With Event Gateway: Hookdeck can filter and route notifications on status or labels in the body, so one contact point can serve several destinations and each gets only the alerts it handles.

Testing and debugging Grafana webhooks

Use the contact point's Test button to prove that Grafana can reach you, then make a real rule fire and resolve to test everything else.

What to know: The Test button sends either a predefined notification or one with your own labels and annotations. It arrives as an ordinary firing notification, with the labels alertname=TestAlert and instance=Grafana by default, and it's signed like any other when HMAC is on. It skips your notification policies, so a passing test doesn't show that real alerts will reach this contact point. It also holds a single alert and never sends resolved. There's no test event type, so make sure your handler doesn't open a real incident for one.

Fix it in your setup:

  1. Send a test notification, then check the status on the Contact points tab.
  2. Point a real rule at the contact point, make it fire, then let it resolve. Expect to wait for the group wait, 30 seconds by default, and for the next group interval.
  3. Cover a group with several alerts, a firing notification that carries a resolved alert, and an alert that fires again with the same labels.
  4. Test verification with a missing signature, a tampered body and a wrong secret, and keep fixtures from Grafana webhook events.

With Event Gateway: The Hookdeck CLI receives webhooks on your machine through a public URL, which Grafana Cloud needs to reach a local server, and you can replay past events instead of waiting for a rule to fire again.

Other Grafana webhook limits

Max Alerts. This setting caps the alerts in one notification, and the docs say any over the limit are ignored. truncatedAlerts tells you how many were left out, so check it if you set a cap. The default, 0, means no limit.

Custom Payload. This option replaces the whole body with your template's output, and Grafana then ignores the Title and Message fields. Your template decides whether status or alerts exist, and the output doesn't have to be JSON. Grafana's docs say it's not yet generally available in Grafana Cloud.

Headers. Extra Headers lets you add headers such as X-API-Key and override Content-Type. It can't set restricted ones, including Authorization, Host, User-Agent and Content-Length. Use Basic Authentication, the Authorization header settings or OAuth2 for credentials. You can use Basic Authentication or the Authorization header, not both.

Secrets in the URL. Grafana's docs warn that the URL is stored in plain text and isn't masked when read back, so don't put tokens in it.

Signing needs Grafana 12.0. Earlier versions have no HMAC option, so use Basic Authentication or the Authorization header there, which authenticate the sender but don't protect the body.

Fields beyond the docs. Real bodies carry fields the published example leaves out, such as valueString, ruleUID and a per-alert orgId, as Grafana's own integration test shows. Ignore fields you don't recognize rather than failing on them.

With Event Gateway: Hookdeck's transformations can reshape a body or add headers on the way through, and its queue delivers at a rate you configure when many alerts fire at once.

Keep an endpoint outage from losing your Grafana alerts.

Hookdeck Event Gateway accepts every Grafana notification straight away, verifies the signature, queues alerts and retries to your service on your schedule, so a slow or failing handler doesn't cost you a resolved notification.

Not affiliated with Grafana. Facts are compiled from public sources and linked inline.