# Grafana Webhooks

Covers the Grafana Alerting webhook contact point (Grafana-managed alerting, the
only alerting system in Grafana 11+; also Grafana Cloud).

Not covered: Grafana legacy dashboard alerting webhooks (removed in Grafana 11 —
different payload with `ruleName`/`evalMatches`, no HMAC); Grafana OnCall / IRM
outgoing webhooks (separate product, own templates and auth); Prometheus
Alertmanager `webhook_config` (payload is a close relative — Grafana's is
Alertmanager's plus extra fields — but Alertmanager has no HMAC signing).

## When to Use This Skill

* How do I receive Grafana alert webhooks?
* How do I verify a Grafana webhook signature?
* What is `X-Grafana-Alerting-Signature` and how do I check it?
* Why is my Grafana webhook HMAC verification failing?
* How do I tell a firing alert from a resolved one in the Grafana payload?
* How do I secure a Grafana webhook contact point (HMAC, basic auth, bearer token)?

## Verification (core)

HMAC-SHA256, lowercase hex, over the raw body — or over
`<unix-seconds> + ":" + rawBody` when a timestamp header is configured. The digest is
bare: no `sha256=` prefix, no `t=...,v1=` structure. Both header names are
user-configurable; the signature header defaults to `X-Grafana-Alerting-Signature`
and the timestamp header has no default name (unset = body-only signing).

Node:

```javascript
const crypto = require('crypto');

function verify(rawBody, signature, timestamp, secret) {
  if (!signature || !secret) return false;
  const hmac = crypto.createHmac('sha256', secret);       // secret used as-is (UTF-8)
  if (timestamp) hmac.update(`${timestamp}:`);            // COLON separator, seconds
  hmac.update(rawBody);                                   // RAW bytes, never re-serialized JSON
  const expected = hmac.digest('hex');
  const received = Buffer.from(signature.toLowerCase(), 'utf8');
  const want = Buffer.from(expected, 'utf8');
  return received.length === want.length && crypto.timingSafeEqual(received, want);
}

```

Python:

```python
import hmac, hashlib

def verify(raw_body: bytes, signature: str, timestamp: str | None, secret: str) -> bool:
    if not signature or not secret:
        return False
    mac = hmac.new(secret.encode("utf-8"), digestmod=hashlib.sha256)
    if timestamp:
        mac.update(f"{timestamp}:".encode("utf-8"))   # COLON separator, unix seconds
    mac.update(raw_body)                              # RAW bytes
    # Compare bytes: compare_digest raises TypeError on non-ASCII str input
    received = signature.strip().lower().encode("utf-8", errors="replace")
    return hmac.compare_digest(received, mac.hexdigest().encode("ascii"))

```

If you configure a timestamp header, also reject stale timestamps (a replay window is
your choice — Grafana documents no tolerance; 300s is a sane default) and reject
requests that arrive without the header.

> For complete handlers with route wiring, dispatch, and tests, see:
> 
> * [examples/express/](https://github.com/hookdeck/webhook-skills/tree/main/skills/grafana-webhooks/examples/express/)
> * [examples/nextjs/](https://github.com/hookdeck/webhook-skills/tree/main/skills/grafana-webhooks/examples/nextjs/)
> * [examples/fastapi/](https://github.com/hookdeck/webhook-skills/tree/main/skills/grafana-webhooks/examples/fastapi/)

## No Event Types — Dispatch on Status

Grafana sends no event-type header and no event-type field. Each request is one
notification for an alert group. Dispatch on these fields instead:

| Field | Values | Meaning |
| --- | --- | --- |
| `status` | `firing`, `resolved` | Group status — `firing` if any alert in the group is firing |
| `state` | `alerting`, `ok` | Grafana's equivalent of `status` |
| `alerts[].status` | `firing`, `resolved` | Per-alert status |

A `firing` notification can contain individual `resolved` alerts, so iterate
`alerts[]` rather than trusting the top-level `status` alone. Resolved notifications
can be turned off with Disable resolved message.

## Payload Shape (default)

```json
{
  "receiver": "My Super Webhook",
  "status": "firing",
  "orgId": 1,
  "alerts": [
    {
      "status": "firing",
      "labels": { "alertname": "High memory usage", "team": "blue", "zone": "us-1" },
      "annotations": { "description": "The system has high memory usage" },
      "startsAt": "2021-10-12T09:51:03.157076+02:00",
      "endsAt": "0001-01-01T00:00:00Z",
      "generatorURL": "https://play.grafana.org/alerting/1afz29v7z/edit",
      "fingerprint": "c6eadffa33fcdf37",
      "silenceURL": "https://play.grafana.org/alerting/silence/new?...",
      "dashboardURL": "",
      "panelURL": "",
      "values": { "B": 44.23943737541908, "C": 1 }
    }
  ],
  "groupLabels": {},
  "commonLabels": { "team": "blue" },
  "commonAnnotations": {},
  "externalURL": "https://play.grafana.org/",
  "version": "1",
  "groupKey": "{}:{}",
  "truncatedAlerts": 0,
  "title": "[FIRING:2]  (blue)",
  "state": "alerting",
  "message": "**Firing**\n..."
}

```

`endsAt` is `"0001-01-01T00:00:00Z"` while an alert is still firing. `title` and
`message` are template-driven. `truncatedAlerts` counts alerts dropped by Max
Alerts. With the Custom Payload option the body is whatever the template renders
— possibly pretty-printed or not JSON at all — which is exactly why you must sign and
verify the raw bytes.

## Important Headers

| Header | Description |
| --- | --- |
| `X-Grafana-Alerting-Signature` | Default name for the HMAC-SHA256 hex digest. User-configurable — read the name from config. |
| (your chosen name) | Unix-seconds timestamp, only if you set Timestamp Header. No default name; docs' example uses `X-Grafana-Alerting-Signature-Timestamp`. |
| `Content-Type` | `application/json` by default; overridable via Extra Headers. |

There is no delivery-id header (`X-Grafana-Delivery`, `X-Grafana-Event` etc. do
not exist — don't invent them). Self-hosted Grafana sends from your own egress IPs;
Grafana Cloud publishes source-IP lists (Hosted Grafana for Grafana-managed alerts:
`https://grafana.com/api/hosted-grafana/source-ips.txt`). Those IPs are shared by all
Grafana Cloud customers, so treat an allowlist as defence in depth, not verification.

## HTTP Behaviour

* Method `POST` by default; `PUT` is selectable (`httpMethod`).
* Any 2xx counts as success.
* No handshake / challenge request. The contact point's Test button sends a
  normal, signed notification with a synthetic alert (labels `alertname: TestAlert`,
  `instance: Grafana`; annotation `summary: Notification test`).

## Other Auth Options

Configured on the same contact point, and combinable with HMAC — but only HMAC proves
payload integrity; the rest only authenticate the sender. Compare credentials in
constant time too.

| Option | Sends |
| --- | --- |
| HTTP Basic Authentication | `Authorization: Basic base64(user:pass)` |
| Authorization Header (`authorization_scheme`, default `Bearer`, + `authorization_credentials`) | `Authorization: <scheme> <credentials>` |
| TLS client certificate (mTLS) | Client cert on the TLS handshake |

Grafana rejects a config that sets both Basic auth and the Authorization header
("both HTTP Basic Authentication and Authorization Header are set, only 1 is
permitted"). Recent Grafana versions also expose an HTTP-client subform (OAuth2 client
credentials, proxy). Extra Headers adds static headers, but `Authorization`,
`User-Agent`, `Host` and similar are restricted.

## Idempotency

There is no delivery id. If you need a dedupe key, derive a heuristic one from
`groupKey` + `status` + the sorted `alerts[].fingerprint` and `alerts[].startsAt`
values. Treat it as a heuristic, not a guarantee.

## Environment Variables

```bash
GRAFANA_WEBHOOK_SECRET=your_hmac_secret          # Contact point → HMAC Signature → Secret
GRAFANA_SIGNATURE_HEADER=X-Grafana-Alerting-Signature   # Optional; this is the default
GRAFANA_TIMESTAMP_HEADER=                        # Optional; empty = body-only signing
GRAFANA_MAX_AGE_SECONDS=300                      # Our replay window, not Grafana's

```

HMAC is optional in Grafana (off until you fill the HMAC Signature subform), but a
receiver should require it: if `GRAFANA_WEBHOOK_SECRET` is unset, fail closed.

## Local Development

```bash
# Start tunnel (no account needed)
npx hookdeck-cli listen 3000 grafana --path /webhooks/grafana

```

## Reference Materials

* [references/overview.md](https://github.com/hookdeck/webhook-skills/blob/main/skills/grafana-webhooks/references/overview.md) - What Grafana Alerting webhooks are, payload and statuses
* [references/setup.md](https://github.com/hookdeck/webhook-skills/blob/main/skills/grafana-webhooks/references/setup.md) - Create the contact point, enable HMAC, provisioning YAML
* [references/verification.md](https://github.com/hookdeck/webhook-skills/blob/main/skills/grafana-webhooks/references/verification.md) - Signature verification details and gotchas