Grafana webhooks

Grafana webhooks are HTTP requests that a Webhook contact point in Grafana Alerting sends to your URL when a group of alerts fires or resolves.

2 events covered

These pages cover the Webhook integration on a Grafana Alerting contact point. Grafana IRM has its own outgoing webhooks, with separate templates and authentication.

How Grafana webhooks work

Webhooks push alerts to your system as their state changes, so you don't have to poll Grafana for them. A notification policy groups alerts by label and routes each group to a contact point, and each group's notification is one HTTP request with a JSON body, POST by default. By default Grafana sends the first notification 30 seconds after a group appears, then at most every 5 minutes while the group changes, and repeats an unchanged group every 4 hours.

Grafana has no event types. The top-level status field is firing if any alert in the group is firing and resolved once all of them have resolved, and each entry in alerts[] has its own status, so a firing notification can carry resolved alerts. version is the payload version, currently "1". The Custom Payload option replaces this body with whatever your template renders.

Key facts

Signature header
X-Grafana-Alerting-Signature

Not every delivery is signed: check your setup.

Scheme
HMAC-SHA256, hex
Signed material
{raw body}

The request body as sent. If you set a timestamp header on the contact point, Grafana signs {unix seconds}:{raw body} instead: that header's value, a colon, then the body.

How events are identified
status field in the body (firing or resolved); no event types
Body format
JSON

Content type: application/json.

API versioning
version in the body

Seen: 1.

Envelope
receiver, status, orgId, alerts, groupLabels, commonLabels, commonAnnotations, externalURL, version, groupKey, truncatedAlerts, title, state, message
Agent skill
npx skills add hookdeck/webhook-skills --skill grafana-webhooks

Verify Grafana webhooks →

Setting up Grafana webhooks

  • Create a contact point under Alerts & IRM > Alerting > Notification configuration > Contact points, choose the Webhook integration and enter your URL. You can also create one in a provisioning file. Then assign it to a notification policy or an alert rule, or it won't receive anything.
  • Test it with the Test button on the contact point, which sends a firing notification with a synthetic alert. To reach your own machine from Grafana Cloud you'll need a public URL, such as a tunnel.
  • Check results on the Contact points tab, which shows the status of the latest deliveries, and on the History page's Notifications tab. Grafana OSS and Enterprise need Loki for that history.
  • Grafana has no way to resend a notification by hand. A failed one is tried again at the group's next interval, as long as the group still has alerts.

Delivery behaviour

  1. Repeated notifications

    Grafana resends an unchanged group every 4 hours by default, and its high availability setup can send duplicates. Skip an alert you've already processed, using fingerprint, startsAt and status as the key.

    Handle duplicates →
  2. A resolved notification never arrives

    If your endpoint keeps failing, Grafana drops resolved alerts after 3 group intervals, 15 minutes by default, so you never hear they ended. Expire open incidents on your side rather than waiting for resolved.

    Retries and lost notifications →
  3. Timeouts after 30 seconds

    Grafana abandons a request after 30 seconds and you can't change that on a contact point. Return a 2xx first, because a slow handler that finishes its work still gets the group sent again.

    Timeouts →
  4. Out-of-order or mixed notifications

    Grafana's docs say high availability can cause out-of-order notifications, and a firing body can hold resolved alerts. Read each alert's status, and compare startsAt with what you've stored for that fingerprint.

    Handle ordering →

Grafana webhook best practices

Common Grafana webhook events

Alert group status

firing At least one alert in the group is firing
resolved Every alert in the group has resolved

Events

Resource Events Payloads
Grafana events

firing · resolved

2 1

All 2 events →

In this hub

Webhook events → Every covered Grafana event, with payloads where the registry has them.
MCP events → How an agent subscribes to Grafana events as MCP events.
Verify signatures → Signature facts, verification code and common failures.
Production guide → Production problems: timeouts, duplicates, ordering and retries, and how to handle them.
Agent skill → Install the webhook skill in your coding agent. npx skills add hookdeck/webhook-skills --skill grafana-webhooks

Keep an endpoint outage from losing your Grafana alerts.

Hookdeck Event Gateway accepts every Grafana notification straight away, verifies the signature, queues alerts and retries to your service on your schedule, so a slow or failing handler doesn't cost you a resolved notification.

Not affiliated with Grafana. Facts are compiled from public sources and linked inline.