Guide to Grafana Webhooks Features and Best Practices

Grafana has become the de facto standard for observability dashboards, used by organizations of all sizes to visualize metrics, logs, and traces. Beyond visualization, Grafana's alerting system enables teams to respond to incidents in real-time, and webhooks are the most flexible way to integrate those alerts with external systems.

This guide covers everything you need to know about Grafana webhooks: their features, how to configure them, best practices for production deployments, and the common pain points developers face along with solutions to address them. If you're experiencing timeout issues, see our guide on how to solve Grafana webhook timeout errors.

What are Grafana webhooks?

Grafana webhooks are HTTP callbacks that deliver alert notifications to external endpoints whenever alert conditions change. When an alert rule fires, resolves, or changes state, Grafana sends a JSON payload containing alert details to URLs you configure. This enables integration with incident management systems, custom automation, ticketing platforms, and any service that can receive HTTP requests.

Webhooks in Grafana exist in two primary contexts:

1. Alerting Contact Points - The primary webhook integration for Grafana's unified alerting system
2. Grafana OnCall Outgoing Webhooks - Webhooks triggered by OnCall events for incident response automation

This guide focuses primarily on alerting webhooks, as they're the most commonly used webhook integration.

Grafana webhook features

Supported alert states

Grafana webhooks can notify you of the following alert state changes:

Each webhook delivery includes the current status, allowing your endpoint to take appropriate action based on whether alerts are starting or ending.

Webhook payload structure

When Grafana sends a webhook notification, it delivers a comprehensive JSON payload with the following structure:

{
  "receiver": "webhook-receiver",
  "status": "firing",
  "orgId": 1,
  "alerts": [
    {
      "status": "firing",
      "labels": {
        "alertname": "HighCPUUsage",
        "instance": "server-01",
        "severity": "critical"
      },
      "annotations": {
        "summary": "CPU usage is above 90%",
        "description": "Server server-01 has CPU usage of 95%",
        "runbook_url": "https://wiki.example.com/runbooks/high-cpu"
      },
      "startsAt": "2025-01-21T10:30:00.000Z",
      "endsAt": "0001-01-01T00:00:00Z",
      "generatorURL": "https://grafana.example.com/alerting/grafana/abc123/view",
      "fingerprint": "a1b2c3d4e5f6",
      "silenceURL": "https://grafana.example.com/alerting/silence/new?alertmanager=grafana&matcher=alertname%3DHighCPUUsage",
      "dashboardURL": "https://grafana.example.com/d/abc123",
      "panelURL": "https://grafana.example.com/d/abc123?viewPanel=1",
      "values": {
        "B": 95.2,
        "C": 1
      }
    }
  ],
  "groupLabels": {
    "alertname": "HighCPUUsage"
  },
  "commonLabels": {
    "alertname": "HighCPUUsage",
    "severity": "critical"
  },
  "commonAnnotations": {
    "summary": "CPU usage is above 90%"
  },
  "externalURL": "https://grafana.example.com",
  "version": "1",
  "groupKey": "{}:{alertname=\"HighCPUUsage\"}",
  "truncatedAlerts": 0,
  "title": "[FIRING:1] HighCPUUsage",
  "state": "alerting",
  "message": "CPU usage is above 90%"
}

Key payload fields

Security with HMAC signatures

Grafana supports HMAC-SHA256 signatures to verify webhook authenticity. When enabled, Grafana signs the payload with a shared secret, and the signature is included in the X-Grafana-Alerting-Signature header.

You can also configure a timestamp header to prevent replay attacks by verifying the request was generated recently.

Note: HMAC signature support was added in Grafana 12.0 (May 2025). Users on earlier versions should use Basic Auth or implement verification at their endpoint.

Authentication options

Grafana webhooks support multiple authentication methods:

Note: You can configure either Basic Authentication or the Authorization header, but not both simultaneously.

Alert grouping

Grafana groups related alerts together before sending webhooks, reducing notification noise. Grouping is configured via notification policies using labels like alertname, cluster, or custom labels. A single webhook delivery may contain multiple alerts that share the same group labels.

Custom payload templates

Grafana allows customisation of webhook payloads using Go templates. You can:

• Add custom key-value pairs to the default payload using the Title and Message fields
• Use notification templates to format alert data
• Access alert labels, annotations, and values in templates

The Custom Payload feature (currently in preview) enables complete control over the payload structure, though when enabled, the Title and Message fields are ignored.

Setting up Grafana webhooks

Via the Grafana UI

1. Navigate to Alerting → Contact points
2. Click + Add contact point
3. Enter a descriptive name (e.g., "Incident Management Webhook")
4. From the Integration dropdown, select Webhook
5. Configure the webhook settings:
   • URL: Your HTTPS endpoint that will receive webhooks
   • HTTP Method: POST (default) or PUT
   • Basic Auth: Username and password if required
   • Authorization Header: Scheme and credentials if using token auth
   • HMAC Secret: Secret key for signature verification
6. Optionally configure:
   • Title: Custom title template
   • Message: Custom message template
7. Click Test to verify connectivity
8. Click Save contact point

Via provisioning (YAML)

For infrastructure-as-code deployments, configure webhooks via provisioning files:

# /etc/grafana/provisioning/alerting/contact-points.yaml
apiVersion: 1
contactPoints:
  - orgId: 1
    name: incident-webhook
    receivers:
      - uid: webhook-1
        type: webhook
        settings:
          url: https://your-endpoint.com/webhooks/grafana
          httpMethod: POST
          username: webhook-user
        secureSettings:
          password: ${WEBHOOK_PASSWORD}  # Use secureSettings for sensitive values

Via the HTTP API

curl -X POST https://grafana.example.com/api/v1/provisioning/contact-points \
  -H "Authorization: Bearer $GRAFANA_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "incident-webhook",
    "type": "webhook",
    "settings": {
      "url": "https://your-endpoint.com/webhooks/grafana",
      "httpMethod": "POST"
    }
  }'

Linking to notification policies

After creating a contact point, link it to a notification policy to determine which alerts trigger webhooks:

1. Navigate to Alerting → Notification policies
2. Edit the default policy or create a nested policy
3. Set the Contact point to your webhook
4. Configure Grouping options (group_by labels, group_wait, group_interval)
5. Save the policy

Best practices when working with Grafana webhooks

Verifying HMAC signatures

When processing webhooks from Grafana, verify the HMAC signature to ensure requests genuinely originated from your Grafana instance.

Node.js

const crypto = require('crypto');
const express = require('express');
const app = express();

// Capture raw body for signature verification
app.use(express.json({
  verify: (req, res, buf) => {
    req.rawBody = buf.toString('utf8');
  }
}));

function verifyGrafanaSignature(rawBody, signature, secret, timestamp = null) {
  // If timestamp is provided, include it in the signature calculation
  const payload = timestamp ? `${timestamp}.${rawBody}` : rawBody;

  const hash = crypto
    .createHmac('sha256', secret)
    .update(payload, 'utf8')
    .digest('hex');

  return crypto.timingSafeEqual(
    Buffer.from(hash),
    Buffer.from(signature)
  );
}

app.post('/webhooks/grafana', (req, res) => {
  const signature = req.headers['x-grafana-alerting-signature'];
  const timestamp = req.headers['x-grafana-alerting-timestamp'];

  if (!verifyGrafanaSignature(req.rawBody, signature, process.env.GRAFANA_WEBHOOK_SECRET, timestamp)) {
    return res.status(401).send('Invalid signature');
  }

  // Optionally verify timestamp is recent (prevent replay attacks)
  if (timestamp) {
    const requestTime = parseInt(timestamp, 10) * 1000;
    const now = Date.now();
    if (Math.abs(now - requestTime) > 300000) { // 5 minutes
      return res.status(401).send('Request too old');
    }
  }

  // Process webhook
  res.status(200).send('OK');
});

Python

import hmac
import hashlib
import time
import os
from flask import Flask, request, abort

app = Flask(__name__)

def verify_grafana_signature(payload, signature, secret, timestamp=None):
    if timestamp:
        message = f"{timestamp}.{payload}"
    else:
        message = payload

    computed = hmac.new(
        secret.encode('utf-8'),
        message.encode('utf-8'),
        hashlib.sha256
    ).hexdigest()

    return hmac.compare_digest(computed, signature)

@app.route('/webhooks/grafana', methods=['POST'])
def handle_grafana_webhook():
    signature = request.headers.get('X-Grafana-Alerting-Signature')
    timestamp = request.headers.get('X-Grafana-Alerting-Timestamp')

    if not verify_grafana_signature(
        request.get_data(as_text=True),
        signature,
        os.environ['GRAFANA_WEBHOOK_SECRET'],
        timestamp
    ):
        abort(401)

    # Process webhook
    return 'OK', 200

Decouple webhook processing to avoid timeouts

Grafana has a 30-second timeout for webhook deliveries. If your endpoint takes longer to respond, Grafana considers the delivery failed, so you'll need to process webhooks asynchronously.

Use the fingerprint for idempotency

Each alert includes a fingerprint field—a hash of the alert's labels that uniquely identifies the alert instance. Use this to implement idempotent processing:

async function processGrafanaAlert(alert, groupKey) {
  const idempotencyKey = `${groupKey}-${alert.fingerprint}-${alert.status}`;

  // Check if already processed
  const exists = await redis.get(`processed:${idempotencyKey}`);
  if (exists) {
    console.log(`Alert ${idempotencyKey} already processed, skipping`);
    return;
  }

  // Process the alert
  await handleAlert(alert);

  // Mark as processed with TTL
  await redis.setex(`processed:${idempotencyKey}`, 86400, '1'); // 24 hour TTL
}

Handle alert groups correctly

A single webhook may contain multiple alerts. Process each alert individually while respecting the group context:

async function handleGrafanaWebhook(payload) {
  const { status, alerts, groupLabels, commonLabels } = payload;

  console.log(`Received ${alerts.length} alerts with status: ${status}`);
  console.log(`Group labels: ${JSON.stringify(groupLabels)}`);

  for (const alert of alerts) {
    await processAlert(alert, {
      groupStatus: status,
      groupLabels,
      commonLabels
    });
  }
}

Grafana webhook limitations and pain points

Fixed 30-second timeout

The Problem: Grafana webhooks have a hardcoded 30-second timeout that cannot be configured through the UI. For endpoints that require longer processing times or when delivering large batches of alerts, this can cause failures.

Why It Happens: The timeout is set at the Grafana server level and isn't exposed in the contact point configuration.

Workarounds:

• Always acknowledge webhooks immediately and process asynchronously
• For self-hosted Grafana, you may be able to adjust server-level timeout settings
• If using Prometheus Alertmanager directly, it supports configurable webhook timeouts

How Hookdeck Can Help: Hookdeck provides configurable delivery timeouts up to 60 seconds or more, giving your endpoint additional time to process complex alert batches without risking failed deliveries.

Limited custom header support

The Problem: Grafana webhooks don't support arbitrary custom headers. You can configure Basic Auth or an Authorization header, but not custom headers like X-API-Key, X-Tenant-ID, or Content-Type variants.

Impact: Integrating with APIs that require specific headers often requires an intermediary service or workarounds.

Workarounds:

• Use Basic Auth credentials creatively (some systems accept API keys as the username)
• Deploy a lightweight proxy that adds required headers before forwarding
• Use the Authorization header with custom schemes where supported

How Hookdeck Can Help: Hookdeck's transformations allow you to add custom headers to requests before forwarding them to your endpoint, eliminating the need for a custom proxy.

Custom payload still in preview

The Problem: The Custom Payload feature that allows complete control over the webhook body structure is not yet generally available in Grafana Cloud and may have limitations in some Grafana versions.

Impact: Integrating with systems that expect specific payload formats requires post-processing or an intermediary transformation layer.

Workarounds:

• Use the Title and Message fields with templates for basic customisation
• Build a transformation service that receives Grafana's standard payload and reformats it
• Wait for Custom Payload to become generally available

How Hookdeck Can Help: Hookdeck's transformation capabilities allow you to reshape webhook payloads in-flight, converting Grafana's standard format to match your target system's requirements without building custom infrastructure.

Duplicate alerts in HA deployments

The Problem: In high-availability Grafana deployments, alerts can fire from multiple instances, resulting in duplicate webhook deliveries. This is a known limitation when Grafana instances aren't properly synchronised.

Why It Happens: Without proper HA gossip configuration, each Grafana instance independently evaluates alerts and sends notifications.

Workarounds:

• Configure Grafana HA properly with peer discovery (ha_peer and ha_listen_address)
• Implement idempotency in your webhook handler using fingerprints
• Use external Alertmanager with HA clustering instead of Grafana's built-in alerting

How Hookdeck Can Help: Hookdeck's deduplication feature can automatically filter duplicate webhooks based on payload content or custom identifiers, ensuring your endpoint only processes each unique alert once.

Aggressive retry behaviour

The Problem: When webhooks fail, Grafana can generate aggressive retry attempts that may overwhelm both your endpoint and Grafana itself. Users have reported waves of retries causing system instability.

Impact: Temporary endpoint issues can escalate into broader system problems.

Workarounds:

• Ensure your endpoint returns appropriate HTTP status codes (4xx for client errors to stop retries)
• Monitor Grafana logs for retry storms
• Implement circuit breakers in your webhook handler

How Hookdeck Can Help: Hookdeck provides configurable retry policies with exponential backoff, preventing retry storms while ensuring reliable delivery. Failed webhooks are preserved for manual replay.

Limited visibility into delivery status

The Problem: Grafana provides minimal visibility into webhook delivery success or failure. The alerting UI shows alert states but not delivery status, making it difficult to know if notifications actually reached their destinations.

Impact: You may not realise webhooks are failing until incidents go unnoticed.

Workarounds:

• Monitor Grafana server logs for delivery errors
• Implement acknowledgment tracking in your webhook handler
• Create alerts that monitor your webhook endpoint's health

How Hookdeck Can Help: Hookdeck's dashboard provides complete visibility into webhook delivery status, latency, errors, and response codes. Configure alerts to notify you when delivery issues occur.

No built-in dead letter queue

The Problem: Webhooks that fail after all retries are exhausted are simply lost. Grafana doesn't maintain a record of failed deliveries for later inspection or replay.

Impact: During endpoint outages, alerts may be permanently lost with no way to recover them.

Workarounds:

• Ensure your endpoint has high availability
• Implement your own logging of received webhooks for gap analysis
• Use a queuing service between Grafana and your final endpoint

How Hookdeck Can Help: Hookdeck automatically preserves failed webhooks in a dead letter queue, allowing you to inspect, debug, and replay them once issues are resolved.

Testing Grafana webhooks

Use the built-in test feature

The contact point configuration includes a Test button that sends a sample payload. However, be aware that:

• Test payloads may differ slightly from real alerts
• The test may succeed while real deliveries fail due to payload size differences
• Some authentication issues only surface with actual alert data

Use a request inspector

Before building your handler, inspect real Grafana payloads using services like Hookdeck's Console:

1. Create a temporary URL
2. Configure it as your webhook endpoint
3. Trigger a real alert by violating an alert condition
4. Inspect the payload structure and headers

Validate in staging

Test your webhook integration with realistic scenarios:

• Single alert firing and resolving
• Multiple alerts in a group
• High-cardinality alerts (many label combinations)
• Rapid state changes

Conclusion

Grafana webhooks provide a flexible foundation for integrating your observability platform with incident management, automation, and custom systems. The rich payload structure with labels, annotations, and evaluated values enables sophisticated alert routing and processing.

However, limitations around timeouts, custom headers, payload customisation, and delivery visibility mean production deployments require careful consideration. Implementing proper signature verification, idempotent processing, and asynchronous handling will address most common issues.

For most deployments with reliable endpoints and moderate alert volumes, Grafana's built-in webhook integration combined with proper error handling and idempotency works well. For high-volume alerting systems, complex multi-destination routing, or mission-critical incident response workflows where delivery guarantees matter, webhook infrastructure like Hookdeck can address Grafana's limitations, providing you with configurable timeouts, payload transformation, automatic deduplication, and comprehensive delivery monitoring without modifying your Grafana configuration.