Grafana Webhooks Skill

Covers the Grafana Alerting webhook contact point (Grafana-managed alerting, the only alerting system in Grafana 11+; also Grafana Cloud).

Not covered: Grafana legacy dashboard alerting webhooks (removed in Grafana 11 — different payload with ruleName/evalMatches, no HMAC); Grafana OnCall / IRM outgoing webhooks (separate product, own templates and auth); Prometheus Alertmanager webhook_config (payload is a close relative — Grafana's is Alertmanager's plus extra fields — but Alertmanager has no HMAC signing).

When to Use This Skill

• How do I receive Grafana alert webhooks?
• How do I verify a Grafana webhook signature?
• What is X-Grafana-Alerting-Signature and how do I check it?
• Why is my Grafana webhook HMAC verification failing?
• How do I tell a firing alert from a resolved one in the Grafana payload?
• How do I secure a Grafana webhook contact point (HMAC, basic auth, bearer token)?

Verification (core)

HMAC-SHA256, lowercase hex, over the raw body — or over <unix-seconds> + ":" + rawBody when a timestamp header is configured. The digest is bare: no sha256= prefix, no t=...,v1= structure. Both header names are user-configurable; the signature header defaults to X-Grafana-Alerting-Signature and the timestamp header has no default name (unset = body-only signing).

Node:

const crypto = require('crypto');

function verify(rawBody, signature, timestamp, secret) {
  if (!signature || !secret) return false;
  const hmac = crypto.createHmac('sha256', secret);       // secret used as-is (UTF-8)
  if (timestamp) hmac.update(`${timestamp}:`);            // COLON separator, seconds
  hmac.update(rawBody);                                   // RAW bytes, never re-serialized JSON
  const expected = hmac.digest('hex');
  const received = Buffer.from(signature.toLowerCase(), 'utf8');
  const want = Buffer.from(expected, 'utf8');
  return received.length === want.length && crypto.timingSafeEqual(received, want);
}

Python:

import hmac, hashlib

def verify(raw_body: bytes, signature: str, timestamp: str | None, secret: str) -> bool:
    if not signature or not secret:
        return False
    mac = hmac.new(secret.encode("utf-8"), digestmod=hashlib.sha256)
    if timestamp:
        mac.update(f"{timestamp}:".encode("utf-8"))   # COLON separator, unix seconds
    mac.update(raw_body)                              # RAW bytes
    # Compare bytes: compare_digest raises TypeError on non-ASCII str input
    received = signature.strip().lower().encode("utf-8", errors="replace")
    return hmac.compare_digest(received, mac.hexdigest().encode("ascii"))

If you configure a timestamp header, also reject stale timestamps (a replay window is your choice — Grafana documents no tolerance; 300s is a sane default) and reject requests that arrive without the header.

For complete handlers with route wiring, dispatch, and tests, see:

• examples/express/
• examples/nextjs/
• examples/fastapi/

No Event Types — Dispatch on Status

Grafana sends no event-type header and no event-type field. Each request is one notification for an alert group. Dispatch on these fields instead:

A firing notification can contain individual resolved alerts, so iterate alerts[] rather than trusting the top-level status alone. Resolved notifications can be turned off with Disable resolved message.

Payload Shape (default)

{
  "receiver": "My Super Webhook",
  "status": "firing",
  "orgId": 1,
  "alerts": [
    {
      "status": "firing",
      "labels": { "alertname": "High memory usage", "team": "blue", "zone": "us-1" },
      "annotations": { "description": "The system has high memory usage" },
      "startsAt": "2021-10-12T09:51:03.157076+02:00",
      "endsAt": "0001-01-01T00:00:00Z",
      "generatorURL": "https://play.grafana.org/alerting/1afz29v7z/edit",
      "fingerprint": "c6eadffa33fcdf37",
      "silenceURL": "https://play.grafana.org/alerting/silence/new?...",
      "dashboardURL": "",
      "panelURL": "",
      "values": { "B": 44.23943737541908, "C": 1 }
    }
  ],
  "groupLabels": {},
  "commonLabels": { "team": "blue" },
  "commonAnnotations": {},
  "externalURL": "https://play.grafana.org/",
  "version": "1",
  "groupKey": "{}:{}",
  "truncatedAlerts": 0,
  "title": "[FIRING:2]  (blue)",
  "state": "alerting",
  "message": "**Firing**\n..."
}

endsAt is "0001-01-01T00:00:00Z" while an alert is still firing. title and message are template-driven. truncatedAlerts counts alerts dropped by Max Alerts. With the Custom Payload option the body is whatever the template renders — possibly pretty-printed or not JSON at all — which is exactly why you must sign and verify the raw bytes.

Important Headers

There is no delivery-id header (X-Grafana-Delivery, X-Grafana-Event etc. do not exist — don't invent them). Self-hosted Grafana sends from your own egress IPs; Grafana Cloud publishes source-IP lists (Hosted Grafana for Grafana-managed alerts: https://grafana.com/api/hosted-grafana/source-ips.txt). Those IPs are shared by all Grafana Cloud customers, so treat an allowlist as defence in depth, not verification.

HTTP Behaviour

• Method POST by default; PUT is selectable (httpMethod).
• Any 2xx counts as success.
• No handshake / challenge request. The contact point's Test button sends a normal, signed notification with a synthetic alert (labels alertname: TestAlert, instance: Grafana; annotation summary: Notification test).

Other Auth Options

Configured on the same contact point, and combinable with HMAC — but only HMAC proves payload integrity; the rest only authenticate the sender. Compare credentials in constant time too.

Grafana rejects a config that sets both Basic auth and the Authorization header ("both HTTP Basic Authentication and Authorization Header are set, only 1 is permitted"). Recent Grafana versions also expose an HTTP-client subform (OAuth2 client credentials, proxy). Extra Headers adds static headers, but Authorization, User-Agent, Host and similar are restricted.

Idempotency

There is no delivery id. If you need a dedupe key, derive a heuristic one from groupKey + status + the sorted alerts[].fingerprint and alerts[].startsAt values. Treat it as a heuristic, not a guarantee.

Environment Variables

GRAFANA_WEBHOOK_SECRET=your_hmac_secret          # Contact point → HMAC Signature → Secret
GRAFANA_SIGNATURE_HEADER=X-Grafana-Alerting-Signature   # Optional; this is the default
GRAFANA_TIMESTAMP_HEADER=                        # Optional; empty = body-only signing
GRAFANA_MAX_AGE_SECONDS=300                      # Our replay window, not Grafana's

HMAC is optional in Grafana (off until you fill the HMAC Signature subform), but a receiver should require it: if GRAFANA_WEBHOOK_SECRET is unset, fail closed.

Local Development

# Start tunnel (no account needed)
npx hookdeck-cli listen 3000 grafana --path /webhooks/grafana

Reference Materials

• references/overview.md - What Grafana Alerting webhooks are, payload and statuses
• references/setup.md - Create the contact point, enable HMAC, provisioning YAML
• references/verification.md - Signature verification details and gotchas