Grafana webhooks
Grafana webhooks are HTTP requests that a Webhook contact point in Grafana Alerting sends to your URL when a group of alerts fires or resolves.
These pages cover the Webhook integration on a Grafana Alerting contact point. Grafana IRM has its own outgoing webhooks, with separate templates and authentication.
How Grafana webhooks work
Webhooks push alerts to your system as their state changes, so you don't have to poll Grafana for them. A notification policy groups alerts by label and routes each group to a contact point, and each group's notification is one HTTP request with a JSON body, POST by default. By default Grafana sends the first notification 30 seconds after a group appears, then at most every 5 minutes while the group changes, and repeats an unchanged group every 4 hours.
Grafana has no event types. The top-level status field is firing if any alert in the group is firing and resolved once all of them have resolved, and each entry in alerts[] has its own status, so a firing notification can carry resolved alerts. version is the payload version, currently "1". The Custom Payload option replaces this body with whatever your template renders.
Key facts
X-Grafana-Alerting-Signature Not every delivery is signed: check your setup.
{raw body} The request body as sent. If you set a timestamp header on the contact point, Grafana signs {unix seconds}:{raw body} instead: that header's value, a colon, then the body.
status field in the body (firing or resolved); no event types receiver, status, orgId, alerts, groupLabels, commonLabels, commonAnnotations, externalURL, version, groupKey, truncatedAlerts, title, state, message Setting up Grafana webhooks
- Create a contact point under Alerts & IRM > Alerting > Notification configuration > Contact points, choose the Webhook integration and enter your URL. You can also create one in a provisioning file. Then assign it to a notification policy or an alert rule, or it won't receive anything.
- Test it with the Test button on the contact point, which sends a
firingnotification with a synthetic alert. To reach your own machine from Grafana Cloud you'll need a public URL, such as a tunnel. - Check results on the Contact points tab, which shows the status of the latest deliveries, and on the History page's Notifications tab. Grafana OSS and Enterprise need Loki for that history.
- Grafana has no way to resend a notification by hand. A failed one is tried again at the group's next interval, as long as the group still has alerts.
Delivery behaviour
30 seconds fixed in Grafana's HTTP client, with no contact point setting Retries at the next group interval no immediate retry; the default interval is 5 minutes After repeated failures resolved alerts dropped after 3 group intervals; firing alerts keep being tried Duplicates expected unchanged groups repeat every 4 hours by default, and HA can double-send Known failure modes
From Hookdeck's Grafana guide →-
Repeated notifications
Grafana resends an unchanged group every 4 hours by default, and its high availability setup can send duplicates. Skip an alert you've already processed, using
Handle duplicates →fingerprint,startsAtandstatusas the key. -
A resolved notification never arrives
If your endpoint keeps failing, Grafana drops resolved alerts after 3 group intervals, 15 minutes by default, so you never hear they ended. Expire open incidents on your side rather than waiting for
Retries and lost notifications →resolved. -
Timeouts after 30 seconds
Grafana abandons a request after 30 seconds and you can't change that on a contact point. Return a 2xx first, because a slow handler that finishes its work still gets the group sent again.
Timeouts → -
Out-of-order or mixed notifications
Grafana's docs say high availability can cause out-of-order notifications, and a
Handle ordering →firingbody can hold resolved alerts. Read each alert'sstatus, and comparestartsAtwith what you've stored for thatfingerprint.
Grafana webhook best practices
- Turn on signing and require it. Signing is off until you set a secret. Once it's on, reject a request with no signature header, because anyone who finds your URL can send a false all-clear.
- Respond within 30 seconds. The limit is fixed and covers the whole request. Return a 2xx first and do the work in a background job.
- Process each alert, not just the group.
A
firingnotification can carry resolved alerts, so readalerts[].statusrather than the top-levelstatus. - Deduplicate on the fingerprint and start time.
There's no delivery ID, and
fingerprintstays the same each time an alert with those labels fires. Key onfingerprint,startsAtandstatustogether. - Don't wait for a resolved notification. A resolved notification can be dropped during an outage. Expire open incidents that stop receiving the repeat reminder, and check Grafana's active alerts.
- Control volume with notification policies. You can't subscribe to event types. Route by label, set Group by and the timers, and use Max Alerts if bodies get too large.
Common Grafana webhook events
Alert group status
firing | At least one alert in the group is firing |
resolved | Every alert in the group has resolved |
Events
| Resource | Events | Payloads |
|---|---|---|
Grafana events | 2 | 1 |
In this hub
| Webhook events → | Every covered Grafana event, with payloads where the registry has them. |
| MCP events → | How an agent subscribes to Grafana events as MCP events. |
| Verify signatures → | Signature facts, verification code and common failures. |
| Production guide → | Production problems: timeouts, duplicates, ordering and retries, and how to handle them. |
| Agent skill → | Install the webhook skill in your coding agent. npx skills add hookdeck/webhook-skills --skill grafana-webhooks |
Keep an endpoint outage from losing your Grafana alerts.
Hookdeck Event Gateway accepts every Grafana notification straight away, verifies the signature, queues alerts and retries to your service on your schedule, so a slow or failing handler doesn't cost you a resolved notification.