Guide to Prometheus Alertmanager Webhooks: Features and Best Practices

Prometheus Alertmanager decides who hears about an alert, and its webhook receiver lets that be any service you run. When Prometheus fires alerts, Alertmanager groups them, applies silences and inhibition rules, and then sends each group's notification to the receiver its routing tree chooses. A webhook receiver turns that notification into an HTTP request to your own endpoint, so you can open incidents, run automated remediation or feed a tool Alertmanager has no dedicated integration for.

This guide covers how the webhook receiver is configured, how to authenticate its requests, what the version 4 payload contains, how grouping and repeat intervals shape what you receive, and the practices that keep an Alertmanager receiver reliable.

What are Prometheus Alertmanager webhooks?

In alertmanager.yml, a receiver is a named list of notification integrations, and the routing tree decides which receiver handles each group of alerts. A webhook_configs entry in a receiver is the generic integration: it sends an HTTP POST with a JSON body to the URL you configure. The body describes one group of alerts, with a group-level status of firing or resolved and an alerts array holding each alert in the group.

Alertmanager does not sign webhook requests. Authentication comes from the receiver's http_config, the same HTTP client settings every HTTP-based integration uses. It can send an Authorization header from basic_auth (a username and password) or from authorization (a credential whose type defaults to Bearer). It can also fetch an OAuth 2.0 token or add custom headers.

Alertmanager is self-hosted. Requests come from wherever your Alertmanager instances run.

Prometheus Alertmanager webhook features

Feature	Details
Configuration	webhook_configs in a receiver in alertmanager.yml
Method and format	POST, JSON body
Payload	Version "4": one notification per alert group
Status	firing or resolved, on the group and on each alert
Resolved notifications	send_resolved, true by default
Truncation	max_alerts caps alerts per message. truncatedAlerts counts the rest
Timeout	timeout, 0s (none) by default, with no effect above group_interval
Custom body	Optional payload Go template, unvalidated and unsupported by the Alertmanager team
Authentication	http_config: basic_auth, authorization (default type Bearer), oauth2 or http_headers
Signature and timestamp	None
Timing	Route group_wait (30s), group_interval (5m) and repeat_interval (4h) by default
Retries	A failed request can be retried. The schedule is not documented
Source IPs	Your own Alertmanager hosts

The webhook payload

Alertmanager documents the body as a schema rather than an example:

{
  "version": "4",
  "groupKey": <string>,              // key identifying the group of alerts (e.g. to deduplicate)
  "truncatedAlerts": <int>,          // how many alerts have been truncated due to "max_alerts"
  "status": "<resolved|firing>",
  "receiver": <string>,
  "groupLabels": <object>,
  "commonLabels": <object>,
  "commonAnnotations": <object>,
  "externalURL": <string>,           // backlink to the Alertmanager.
  "notification_reason": <string>,   // string represent the reason this notification was generated
  "alerts": [
    {
      "status": "<resolved|firing>",
      "labels": <object>,
      "annotations": <object>,
      "startsAt": "<rfc3339>",
      "endsAt": "<rfc3339>",
      "generatorURL": <string>,      // identifies the entity that caused the alert
      "fingerprint": <string>        // fingerprint to identify the alert
    },
    ...
  ]
}

There is no event type. The two values of status are the closest thing, and they are states, not events. A group's status is firing while any alert in it is firing, and a single notification can contain firing and resolved alerts together, so loop over alerts and use each alert's own status. labels carry alertname, severity and whatever else your alerting rules set, and annotations carry text such as a summary or a runbook link.

If you set payload on the webhook config, Alertmanager sends whatever your Go template renders instead, and none of the fields above are guaranteed. Its documentation warns that it does not validate the result and will not support issues arising from the option.

Inspect Alertmanager webhooks as they arrive. Point a webhook receiver at a Hookdeck Inspect URL to see each notification's headers and body, with no account or setup required.

Setting up Prometheus Alertmanager webhooks

Generate a long random credential first, for example with openssl rand -hex 32. Then add a receiver with a webhook config, and route alerts to it:

route:
  receiver: webhook
  group_by: [alertname, cluster]

receivers:
  - name: webhook
    webhook_configs:
      - url: https://example.com/webhooks/alertmanager
        send_resolved: true
        max_alerts: 50
        http_config:
          authorization:
            type: Bearer
            credentials_file: /etc/alertmanager/webhook-token

To use a username and password instead, replace authorization with:

          basic_auth:
            username: alertmanager
            password_file: /etc/alertmanager/webhook-password

basic_auth and authorization are mutually exclusive, and oauth2 cannot be combined with either. The _file variants keep the secret out of the configuration file. Reload Alertmanager after changing its configuration.

To route through Hookdeck, create a Hookdeck source with the Prometheus Alertmanager type and use its URL as the webhook url. Set basic_auth, or authorization with the default Bearer type, in the webhook receiver's http_config. OAuth 2.0 is not supported. The source's verification offers two methods to match:

• Basic Auth, the default: enter the same Username and Password as basic_auth. Alertmanager sends them with every request, which matters because Hookdeck never answers with a 401 challenge.
• Bearer token: enter only the secret from credentials (or the file) in the Credentials field. Alertmanager sends it as Authorization: Bearer <credentials>. Keep type at its default of Bearer.

Credentials sent through http_headers or an OAuth 2.0 token are not checked by this source type. Verification is optional on the source, but leaving it off means anyone who learns the URL can send alerts. The source accepts POST only, which is the method Alertmanager uses.

Securing Prometheus Alertmanager webhooks

Your receiver reads the Authorization header and checks it against whichever method you configured. For a Bearer credential, match the scheme case-insensitively and compare the rest with your stored secret. For basic_auth, decode the value, split on the first colon, and compare the username and password. Do both comparisons in constant time.

const crypto = require('crypto');
const express = require('express');

const BEARER_SECRET = process.env.ALERTMANAGER_BEARER_SECRET; // set this, or the pair below
const BASIC_USERNAME = process.env.ALERTMANAGER_USERNAME;
const BASIC_PASSWORD = process.env.ALERTMANAGER_PASSWORD;

function safeEqual(a, b) {
  // timingSafeEqual throws on unequal lengths, so check first.
  return a.length === b.length && crypto.timingSafeEqual(a, b);
}

function verifyAuthorization(header) {
  if (typeof header !== 'string') return false;

  const bearer = /^bearer (.+)$/i.exec(header);
  if (bearer) {
    return Boolean(BEARER_SECRET) &&
      safeEqual(Buffer.from(bearer[1], 'utf8'), Buffer.from(BEARER_SECRET, 'utf8'));
  }

  const basic = /^basic\s+([A-Za-z0-9+/]+={0,2})\s*$/i.exec(header);
  if (!basic || !BASIC_USERNAME || !BASIC_PASSWORD) return false;
  const decoded = Buffer.from(basic[1], 'base64');
  const colon = decoded.indexOf(0x3a); // split on the first colon
  if (colon === -1) return false;
  const userOk = safeEqual(decoded.subarray(0, colon), Buffer.from(BASIC_USERNAME, 'utf8'));
  const passOk = safeEqual(decoded.subarray(colon + 1), Buffer.from(BASIC_PASSWORD, 'utf8'));
  return userOk && passOk;
}

const app = express();

app.post('/webhooks/alertmanager', express.raw({ type: '*/*' }), (req, res) => {
  if (!verifyAuthorization(req.get('authorization'))) {
    return res.sendStatus(401);
  }

  let notification;
  try {
    notification = JSON.parse(req.body.toString('utf8'));
  } catch {
    return res.sendStatus(400);
  }

  for (const alert of notification.alerts || []) {
    // alert.status is 'firing' or 'resolved'; key your state on alert.fingerprint.
  }
  res.sendStatus(200);
});

The same check in Python:

import base64
import binascii
import hmac
import os

BEARER_SECRET = os.environ.get("ALERTMANAGER_BEARER_SECRET", "")
BASIC_USERNAME = os.environ.get("ALERTMANAGER_USERNAME", "")
BASIC_PASSWORD = os.environ.get("ALERTMANAGER_PASSWORD", "")


def verify_authorization(header) -> bool:
    if not header:
        return False
    scheme, _, value = header.partition(" ")
    if scheme.lower() == "bearer":
        if not BEARER_SECRET:
            return False
        return hmac.compare_digest(value.encode("utf-8"), BEARER_SECRET.encode("utf-8"))
    if scheme.lower() != "basic" or not BASIC_USERNAME or not BASIC_PASSWORD:
        return False
    try:
        decoded = base64.b64decode(value.strip(), validate=True)
    except (binascii.Error, ValueError):
        return False
    user, sep, password = decoded.partition(b":")
    if not sep:
        return False
    user_ok = hmac.compare_digest(user, BASIC_USERNAME.encode("utf-8"))
    pass_ok = hmac.compare_digest(password, BASIC_PASSWORD.encode("utf-8"))
    return user_ok and pass_ok

Either credential is static. It authenticates the sender, not the body, so a modified payload still passes, and with no timestamp a captured request can be replayed. Serve the endpoint over HTTPS only, keep the credential in a file rather than in alertmanager.yml, and rotate it in Alertmanager and in your receiver together.

Make Alertmanager webhooks production-ready. Hookdeck Event Gateway checks the Basic or Bearer credentials you configure, queues every alert notification durably, and retries delivery to your service on a schedule you control.

Prometheus Alertmanager webhook limitations and pain points

The same alert, again and again

The Problem: Your receiver sees the same firing alert every few hours, or a flurry of notifications for one group.

Why It Happens: Alertmanager notifies per group, not per alert. After group_wait (30 seconds by default) it sends the first notification, then checks every group_interval (5 minutes) for changes, and repeats an unchanged notification after repeat_interval (4 hours). Each notification carries the whole group, so an alert you have already seen is sent again whenever the group changes.

Workarounds:

• Key your state on each alert's fingerprint and status, and upsert rather than insert.
• Use groupKey to identify the group, which the docs suggest for deduplication.
• Tune group_by, group_interval and repeat_interval per route rather than filtering in the receiver.

How Hookdeck Can Help: Hookdeck deduplication can drop repeated notifications before they reach your service, based on fields you choose, such as groupKey, status and the alert fingerprints.

Undocumented retries and no default timeout

The Problem: A slow or failing receiver delays notifications, and you cannot tell from the docs how long Alertmanager keeps trying.

Why It Happens: timeout defaults to 0s, meaning no timeout, and setting it above group_interval has no effect. group_interval also bounds each send, so a notification that takes longer than that is cancelled. The docs say a timed-out request can be retried but do not document the schedule or which status codes count as success.

Workarounds:

• Set timeout to a few seconds below your group_interval.
• Respond with a 2xx as soon as the credentials are checked and the notification is stored.

How Hookdeck Can Help: Hookdeck accepts each notification as soon as it arrives and stores it, then delivers to your service with retries you configure. Issues alert you when deliveries to your service start failing, which matters most when the thing failing is your alert path.

Large groups get cut off

The Problem: A big incident produces a notification so large your receiver rejects it, or one that is missing alerts.

Why It Happens: By default every alert in the group is included. With max_alerts set, the extra alerts are dropped from the message and only counted in truncatedAlerts.

Workarounds:

• Set max_alerts to a size your receiver handles, and treat a non-zero truncatedAlerts as a signal to follow externalURL back to Alertmanager for the full list.
• Group more narrowly with group_by so groups stay small.

How Hookdeck Can Help: Hookdeck stores every notification with its full body, so you can inspect exactly what a large group sent and replay it after fixing your handler.

Only two of the auth methods are checked

The Problem: A receiver configured with oauth2 or a custom header never verifies in Hookdeck's Prometheus Alertmanager source.

Why It Happens: The source checks the Authorization header from basic_auth or from authorization with the Bearer type. An OAuth 2.0 token changes on every refresh, and custom headers are not part of this source type.

Workarounds:

• Use basic_auth or authorization on receivers that point at Hookdeck.
• If you are tied to another method, check it in your own receiver instead.

How Hookdeck Can Help: With a supported method, the source compares the credentials you enter. Hookdeck answers Alertmanager with a 200 either way, but requests without matching credentials are recorded as failed verification and never reach your service.

Best practices

Process each alert, not the group

Loop over alerts and act on each one's status, labels and fingerprint. The group-level status hides resolved alerts inside a firing group.

Keep send_resolved on

Resolved notifications let your receiver close incidents automatically. Without them, you need another way to learn that an alert has stopped.

Keep secrets in files

Use password_file or credentials_file so the credential never appears in alertmanager.yml or in version control.

Avoid custom payloads unless you need them

The version 4 body is documented and stable. A payload template is unvalidated and unsupported, so prefer reshaping the body after it arrives.

Acknowledge fast and process asynchronously

Check the credentials, store the notification, return 200, and do the work afterwards. See why to process webhooks asynchronously.

Conclusion

Alertmanager's webhook receiver sends one version 4 JSON notification per alert group, with no signature and no event type, authenticated by basic_auth or a Bearer authorization header from http_config. Keep the credential in a file, compare it in constant time over HTTPS, process each alert by its own status and fingerprint, expect repeats from grouping and repeat_interval, and set a timeout so a slow receiver cannot hold notifications up.

Hookdeck Event Gateway checks the credentials, keeps every notification, and retries and replays delivery to your service, so a deploy on your side does not cost you an alert.

Get started with Hookdeck for free and handle Prometheus Alertmanager webhooks reliably in minutes.