# Grafana webhooks in production: duplicates, ordering, retries and timeouts

Agent skill: `npx skills add hookdeck/webhook-skills --skill grafana-webhooks` ([details](https://hookdeck.com/webhooks/grafana/skill))

This guide covers the problems the Webhook integration in Grafana Alerting causes in production and how to handle each one. What Grafana sends is on the [Grafana webhooks](/webhooks/grafana) overview, `firing` and `resolved` are on [Grafana webhook events](/webhooks/grafana/events), and signature verification code is on [Verify Grafana webhooks](/webhooks/grafana/verify).

Where Grafana's docs leave something out, this guide follows its current source code, since Grafana is open source: the [webhook notifier](https://github.com/grafana/alerting/blob/main/receivers/webhook/v1/webhook.go), the [HTTP client](https://github.com/grafana/alerting/blob/main/http/tls.go) and [Grafana's Alertmanager fork](https://github.com/grafana/prometheus-alertmanager). It doesn't cover Grafana IRM's outgoing webhooks, which are a separate feature.

## At a glance

| Problem | What to do | With Event Gateway |
| --- | --- | --- |
| [Timeouts](#timeouts) | Return a 2xx within 30 seconds; process in a background job | Accepts the notification at once, then queues it for your service |
| [Duplicates](#duplicates) | Key each alert on `fingerprint`, `startsAt` and `status` | Drops repeats before they reach your handler |
| [Ordering](#ordering) | Read `alerts[].status`; compare `startsAt` for each `fingerprint` | Logs every event and attempt, so you can see arrival order when debugging |
| [Retries](#retries) | Stay available; expire incidents that stop being repeated | Accepts notifications while your service is down and retries them |
| [Choosing alerts](#choosing-events) | Route with notification policies, Group by and the timers | Filters and routes on `status` and labels |
| [Testing](#testing) | Use the Test button, then make a real rule fire and resolve | Forwards to localhost and replays past events |
| [Other limits](#other-limits) | Check `truncatedAlerts`; plan for Custom Payload and header limits | Reshapes bodies and delivers at a rate you set |

## Grafana webhook timeouts and slow handlers {#timeouts}

Grafana gives your endpoint 30 seconds to respond, and you can't change that on a contact point. Return a 2xx first, then do the work in a background job.

**Why it happens:** Grafana's notifier HTTP client [sets a 30-second timeout](https://github.com/grafana/alerting/blob/main/http/tls.go) on the whole request, and the webhook integration has no setting for it. Any 2xx is a success. Anything else, including a timeout, is a failure.

Handlers usually run long because of what they call next, such as a ticketing API or an on-call service, or because of a serverless cold start. One notification can also hold many alerts, so work done per alert grows with the size of the group. A handler that finishes after Grafana has given up still did the work, and Grafana sends the group again at its next interval, so a slow handler produces duplicates too.

**Fix it in your handler:**

1. Verify the signature and return 200 straight away.
2. Push the raw body onto a queue and process it in a worker. See [why to process webhooks asynchronously](/webhooks/guides/why-implement-asynchronous-processing-webhooks).
3. Make calls to other services from the worker, with their own timeouts and retries.

**With Event Gateway:** Hookdeck accepts each Grafana notification immediately, so Grafana records a success and moves on. It then delivers to your service with a timeout you can configure, 60 seconds by default and up to 15 minutes per destination, with its own retries.

## Duplicate Grafana webhooks {#duplicates}

Grafana sends the same alert more than once by design, and there's no delivery ID. Key your side effects on each alert's `fingerprint`, `startsAt` and `status`.

**Why it happens:** There are four common causes:

* **The repeat interval.** Grafana resends a group that hasn't changed as a reminder that its alerts are still firing, [every 4 hours by default](https://grafana.com/docs/grafana/latest/alerting/fundamentals/notifications/group-alert-notifications/).
* **Group changes.** At each group interval, 5 minutes by default, a group that has changed gets a new notification. It lists the group's current alerts, including ones you've already seen.
* **High availability.** Every Grafana instance evaluates every rule by default, and the instances gossip to avoid sending twice. [Grafana describes this as best effort](https://grafana.com/docs/grafana/latest/alerting/set-up/configure-high-availability/): "Alertmanager chooses availability over consistency, which may result in occasional duplicated or out-of-order notifications." If the instances can't reach each other, expect more of them.
* **A slow success.** If your handler finishes after the 30-second limit, Grafana counts a failure and sends the group again.

`fingerprint` alone isn't enough. It's a hash of the alert's labels, so it stays the same every time an alert with those labels fires. Keying on `fingerprint` and `status` would swallow the second incident on the same server. Adding `startsAt` separates them, and a repeat reminder then counts as a duplicate, which is usually what you want.

**Fix it in your handler:**

1. Loop over `alerts[]` and build a key for each alert from `fingerprint`, `startsAt` and `status`.
2. Skip any alert whose key you've already processed. See [how to implement webhook idempotency](/webhooks/guides/implement-webhook-idempotency).
3. On self-hosted high availability, [check your `ha_peers` setup](https://grafana.com/docs/grafana/latest/alerting/set-up/configure-high-availability/) and consider raising `ha_peer_timeout`, 15 seconds by default, if your instances are far apart.

```python
def handle_grafana_notification(body):
    for alert in body["alerts"]:
        key = f'{alert["fingerprint"]}:{alert["startsAt"]}:{alert["status"]}'

        # Already handled this alert in this state
        if not cache.set(f"grafana_alert_{key}", True, nx=True, ex=604800):
            continue

        process(alert)

```

**With Event Gateway:** Hookdeck [deduplicates](/docs/deduplication) events before they reach your endpoint, on the whole payload or on fields you choose, so repeated notifications are filtered at the infrastructure level. It works per notification, so keep the per-alert check if a group can change between sends.

## Out-of-order Grafana webhooks {#ordering}

Grafana doesn't promise an order, and its high availability docs say notifications can arrive out of order. Read each alert's own `status`, and compare `startsAt` with what you've stored for that `fingerprint`.

**Why it happens:** Grafana's high availability docs say Alertmanager prefers sending a notification to keeping its instances consistent. Each group also runs on its own timers, so notifications for different groups have no order between them. For more on the general problem, see [Webhook Ordering: Why It's Hard and How to Handle It](/webhooks/guides/webhook-ordering-why-its-hard-and-how-to-handle-it).

The top-level `status` can mislead you as well. It's `firing` if any alert in the group is firing, so a `firing` notification can list alerts that have already resolved. Each notification describes the group as it stood when Grafana sent it.

**Fix it in your handler:**

1. Act on `alerts[].status`, not the top-level `status`.
2. Store the latest `startsAt` for each `fingerprint`, and ignore an alert older than that.
3. Use `endsAt` to spot a resolved alert. It's `0001-01-01T00:00:00Z` while an alert is firing.

**With Event Gateway:** Hookdeck logs every event and delivery attempt, so when an incident's state looks wrong you can see the order notifications actually arrived in and replay them.

## Grafana webhook retries and lost notifications {#retries}

Grafana doesn't retry a failed webhook straight away. It tries again when the group's next interval comes round, and during a long outage it drops resolved alerts, so stay available and don't depend on `resolved` arriving.

**Why it happens:** When a webhook request fails, the [notifier tells Alertmanager not to retry](https://github.com/grafana/alerting/blob/main/receivers/webhook/v1/webhook.go), so there's one attempt, whatever the status code. Grafana's log shows `notify retry canceled due to unrecoverable error after 1 attempts`. Returning a 4xx rather than a 5xx makes no difference.

A failed notification isn't written to the notification log, so at the next group interval, 5 minutes by default, Grafana finds nothing recorded and sends the group again. That carries on for as long as the group has alerts, and Grafana's docs describe nothing that disables a contact point because it keeps failing. Two things can still be lost:

* **Resolved alerts during an outage.** If notifications keep failing, [Grafana deletes resolved alerts after 3 group intervals](https://github.com/grafana/prometheus-alertmanager/blob/main/dispatch/dispatch.go), 15 minutes by default. An outage longer than that can leave an incident open on your side with no `resolved` to close it.
* **Alerts that came and went.** If a group's `firing` notification never succeeded and its alerts then resolve, [Grafana sends nothing](https://github.com/grafana/prometheus-alertmanager/blob/main/notify/notify.go), because as far as it knows you never heard about them.

Grafana also has no queue you can drain and no way to resend a notification by hand.

**Fix it in your handler:**

1. Keep the endpoint available, or put a queue in front of it.
2. Watch delivery from the [Contact points tab](https://grafana.com/docs/grafana/latest/alerting/configure-notifications/manage-contact-points/) and [notification history](https://grafana.com/docs/grafana/latest/alerting/monitor-status/view-notification-history/). Use a different contact point to alert on failures, since a broken one can't report itself.
3. Treat the repeat reminder as a heartbeat. An incident with no notification for longer than your repeat interval may have ended without you hearing, so check it against Grafana's active alerts.

**With Event Gateway:** Hookdeck acknowledges each notification, so Grafana sees a success even while your service is down. Events wait in a queue, retry on the schedule you configure and can be [replayed](/webhooks/guides/dead-letter-queues-webhook-reliability) once your service recovers.

## Choosing which Grafana alerts to send {#choosing-events}

You can't subscribe to event types, because Grafana has none. A contact point receives what notification policies route to it, so use those to control what reaches your endpoint and how often.

**Why it happens:** Each notification is for one group of alerts, and `firing` and `resolved` are that group's status. A policy matches alerts by label and picks the contact point. [Group by](https://grafana.com/docs/grafana/latest/alerting/fundamentals/notifications/group-alert-notifications/) decides which alerts share a notification, and by default that's by alert rule, using `alertname` and `grafana_folder`.

**Fix it in your setup:**

1. Add a child policy with label matchers, so only the alerts this service handles reach its contact point.
2. Set Group by to match how you handle alerts. Grouping by the special label `...` sends each alert on its own, which also means more requests.
3. Tune group wait, group interval and repeat interval. Their defaults are 30 seconds, 5 minutes and 4 hours.
4. Only turn on Disable resolved message if nothing depends on `resolved`, because Grafana then sends no notification when a group's alerts all end.

**With Event Gateway:** Hookdeck can [filter](/docs/filters) and route notifications on `status` or labels in the body, so one contact point can serve several destinations and each gets only the alerts it handles.

## Testing and debugging Grafana webhooks {#testing}

Use the contact point's Test button to prove that Grafana can reach you, then make a real rule fire and resolve to test everything else.

**What to know:** [The Test button](https://grafana.com/docs/grafana/latest/alerting/configure-notifications/manage-contact-points/) sends either a predefined notification or one with your own labels and annotations. It arrives as an ordinary `firing` notification, with the labels `alertname=TestAlert` and `instance=Grafana` by default, and it's signed like any other when HMAC is on. It skips your notification policies, so a passing test doesn't show that real alerts will reach this contact point. It also holds a single alert and never sends `resolved`. There's no `test` event type, so make sure your handler doesn't open a real incident for one.

**Fix it in your setup:**

1. Send a test notification, then check the status on the Contact points tab.
2. Point a real rule at the contact point, make it fire, then let it resolve. Expect to wait for the group wait, 30 seconds by default, and for the next group interval.
3. Cover a group with several alerts, a `firing` notification that carries a resolved alert, and an alert that fires again with the same labels.
4. Test verification with a missing signature, a tampered body and a wrong secret, and keep fixtures from [Grafana webhook events](/webhooks/grafana/events).

**With Event Gateway:** The [Hookdeck CLI](/docs/cli) receives webhooks on your machine through a public URL, which Grafana Cloud needs to reach a local server, and you can replay past events instead of waiting for a rule to fire again.

## Other Grafana webhook limits {#other-limits}

**Max Alerts.** This setting caps the alerts in one notification, and the docs say any over the limit are ignored. `truncatedAlerts` tells you how many were left out, so check it if you set a cap. The default, `0`, means no limit.

**Custom Payload.** This option replaces the whole body with your template's output, and Grafana then ignores the Title and Message fields. Your template decides whether `status` or `alerts` exist, and the output doesn't have to be JSON. Grafana's docs say it's [not yet generally available in Grafana Cloud](https://grafana.com/docs/grafana/latest/alerting/configure-notifications/manage-contact-points/integrations/webhook-notifier/#custom-payload).

**Headers.** Extra Headers lets you add headers such as `X-API-Key` and override `Content-Type`. It can't set restricted ones, including `Authorization`, `Host`, `User-Agent` and `Content-Length`. Use Basic Authentication, the Authorization header settings or OAuth2 for credentials. You can use Basic Authentication or the Authorization header, not both.

**Secrets in the URL.** Grafana's docs warn that the URL is stored in plain text and isn't masked when read back, so don't put tokens in it.

**Signing needs Grafana 12.0.** Earlier versions have no HMAC option, so use Basic Authentication or the Authorization header there, which authenticate the sender but don't protect the body.

**Fields beyond the docs.** Real bodies carry fields the published example leaves out, such as `valueString`, `ruleUID` and a per-alert `orgId`, as [Grafana's own integration test](https://github.com/grafana/grafana/blob/main/pkg/tests/api/alerting/api_notification_channel_test.go) shows. Ignore fields you don't recognize rather than failing on them.

**With Event Gateway:** Hookdeck's [transformations](/docs/transformations) can reshape a body or add headers on the way through, and its queue delivers at a rate you configure when many alerts fire at once.