# Guide to Prometheus Alertmanager Webhooks: Features and Best Practices

Prometheus Alertmanager decides who hears about an alert, and its webhook receiver lets that be any service you run. When Prometheus fires alerts, Alertmanager groups them, applies silences and inhibition rules, and then sends each group's notification to the receiver its routing tree chooses. A webhook receiver turns that notification into an HTTP request to your own endpoint, so you can open incidents, run automated remediation or feed a tool Alertmanager has no dedicated integration for.

This guide covers how the webhook receiver is configured, how to authenticate its requests, what the version 4 payload contains, how grouping and repeat intervals shape what you receive, and the practices that keep an Alertmanager receiver reliable.

## What are Prometheus Alertmanager webhooks?

In `alertmanager.yml`, a **receiver** is a named list of notification integrations, and the routing tree decides which receiver handles each group of alerts. A `webhook_configs` entry in a receiver is the generic integration: it sends an HTTP `POST` with a JSON body to the URL you configure. The body describes one group of alerts, with a group-level `status` of `firing` or `resolved` and an `alerts` array holding each alert in the group.

Alertmanager does not sign webhook requests. Authentication comes from the receiver's `http_config`, the same HTTP client settings every HTTP-based integration uses. It can send an `Authorization` header from `basic_auth` (a username and password) or from `authorization` (a credential whose type defaults to `Bearer`). It can also fetch an OAuth 2.0 token or add custom headers.

Alertmanager is self-hosted. Requests come from wherever your Alertmanager instances run.

## Prometheus Alertmanager webhook features

| Feature | Details |
| --- | --- |
| Configuration | `webhook_configs` in a receiver in `alertmanager.yml` |
| Method and format | `POST`, JSON body |
| Payload | Version `"4"`: one notification per alert group |
| Status | `firing` or `resolved`, on the group and on each alert |
| Resolved notifications | `send_resolved`, `true` by default |
| Truncation | `max_alerts` caps alerts per message. `truncatedAlerts` counts the rest |
| Timeout | `timeout`, `0s` (none) by default, with no effect above `group_interval` |
| Custom body | Optional `payload` Go template, unvalidated and unsupported by the Alertmanager team |
| Authentication | `http_config`: `basic_auth`, `authorization` (default type `Bearer`), `oauth2` or `http_headers` |
| Signature and timestamp | None |
| Timing | Route `group_wait` (30s), `group_interval` (5m) and `repeat_interval` (4h) by default |
| Retries | A failed request can be retried. The schedule is not documented |
| Source IPs | Your own Alertmanager hosts |

## The webhook payload

Alertmanager documents the body as a schema rather than an example:

```
{
  "version": "4",
  "groupKey": <string>,              // key identifying the group of alerts (e.g. to deduplicate)
  "truncatedAlerts": <int>,          // how many alerts have been truncated due to "max_alerts"
  "status": "<resolved|firing>",
  "receiver": <string>,
  "groupLabels": <object>,
  "commonLabels": <object>,
  "commonAnnotations": <object>,
  "externalURL": <string>,           // backlink to the Alertmanager.
  "notification_reason": <string>,   // string represent the reason this notification was generated
  "alerts": [
    {
      "status": "<resolved|firing>",
      "labels": <object>,
      "annotations": <object>,
      "startsAt": "<rfc3339>",
      "endsAt": "<rfc3339>",
      "generatorURL": <string>,      // identifies the entity that caused the alert
      "fingerprint": <string>        // fingerprint to identify the alert
    },
    ...
  ]
}

```

There is no event type. The two values of `status` are the closest thing, and they are states, not events. A group's `status` is `firing` while any alert in it is firing, and a single notification can contain firing and resolved alerts together, so loop over `alerts` and use each alert's own `status`. `labels` carry `alertname`, `severity` and whatever else your alerting rules set, and `annotations` carry text such as a summary or a runbook link.

If you set `payload` on the webhook config, Alertmanager sends whatever your Go template renders instead, and none of the fields above are guaranteed. Its documentation warns that it does not validate the result and will not support issues arising from the option.

> **Inspect Alertmanager webhooks as they arrive.** Point a webhook receiver at a [Hookdeck Inspect](https://inspect.hookdeck.com) URL to see each notification's headers and body, with no account or setup required.

## Setting up Prometheus Alertmanager webhooks

Generate a long random credential first, for example with `openssl rand -hex 32`. Then add a receiver with a webhook config, and route alerts to it:

```yaml
route:
  receiver: webhook
  group_by: [alertname, cluster]

receivers:
  - name: webhook
    webhook_configs:
      - url: https://example.com/webhooks/alertmanager
        send_resolved: true
        max_alerts: 50
        http_config:
          authorization:
            type: Bearer
            credentials_file: /etc/alertmanager/webhook-token

```

To use a username and password instead, replace `authorization` with:

```yaml
          basic_auth:
            username: alertmanager
            password_file: /etc/alertmanager/webhook-password

```

`basic_auth` and `authorization` are mutually exclusive, and `oauth2` cannot be combined with either. The `_file` variants keep the secret out of the configuration file. Reload Alertmanager after changing its configuration.

To route through Hookdeck, create a Hookdeck source with the **Prometheus Alertmanager** type and use its URL as the webhook `url`. Set `basic_auth`, or `authorization` with the default Bearer type, in the webhook receiver's `http_config`. OAuth 2.0 is not supported. The source's verification offers two methods to match:

* **Basic Auth**, the default: enter the same **Username** and **Password** as `basic_auth`. Alertmanager sends them with every request, which matters because Hookdeck never answers with a `401` challenge.
* **Bearer token**: enter only the secret from `credentials` (or the file) in the **Credentials** field. Alertmanager sends it as `Authorization: Bearer <credentials>`. Keep `type` at its default of `Bearer`.

Credentials sent through `http_headers` or an OAuth 2.0 token are not checked by this source type. Verification is optional on the source, but leaving it off means anyone who learns the URL can send alerts. The source accepts `POST` only, which is the method Alertmanager uses.

## Securing Prometheus Alertmanager webhooks

Your receiver reads the `Authorization` header and checks it against whichever method you configured. For a Bearer credential, match the scheme case-insensitively and compare the rest with your stored secret. For `basic_auth`, decode the value, split on the first colon, and compare the username and password. Do both comparisons in constant time.

```javascript
const crypto = require('crypto');
const express = require('express');

const BEARER_SECRET = process.env.ALERTMANAGER_BEARER_SECRET; // set this, or the pair below
const BASIC_USERNAME = process.env.ALERTMANAGER_USERNAME;
const BASIC_PASSWORD = process.env.ALERTMANAGER_PASSWORD;

function safeEqual(a, b) {
  // timingSafeEqual throws on unequal lengths, so check first.
  return a.length === b.length && crypto.timingSafeEqual(a, b);
}

function verifyAuthorization(header) {
  if (typeof header !== 'string') return false;

  const bearer = /^bearer (.+)$/i.exec(header);
  if (bearer) {
    return Boolean(BEARER_SECRET) &&
      safeEqual(Buffer.from(bearer[1], 'utf8'), Buffer.from(BEARER_SECRET, 'utf8'));
  }

  const basic = /^basic\s+([A-Za-z0-9+/]+={0,2})\s*$/i.exec(header);
  if (!basic || !BASIC_USERNAME || !BASIC_PASSWORD) return false;
  const decoded = Buffer.from(basic[1], 'base64');
  const colon = decoded.indexOf(0x3a); // split on the first colon
  if (colon === -1) return false;
  const userOk = safeEqual(decoded.subarray(0, colon), Buffer.from(BASIC_USERNAME, 'utf8'));
  const passOk = safeEqual(decoded.subarray(colon + 1), Buffer.from(BASIC_PASSWORD, 'utf8'));
  return userOk && passOk;
}

const app = express();

app.post('/webhooks/alertmanager', express.raw({ type: '*/*' }), (req, res) => {
  if (!verifyAuthorization(req.get('authorization'))) {
    return res.sendStatus(401);
  }

  let notification;
  try {
    notification = JSON.parse(req.body.toString('utf8'));
  } catch {
    return res.sendStatus(400);
  }

  for (const alert of notification.alerts || []) {
    // alert.status is 'firing' or 'resolved'; key your state on alert.fingerprint.
  }
  res.sendStatus(200);
});

```

The same check in Python:

```python
import base64
import binascii
import hmac
import os

BEARER_SECRET = os.environ.get("ALERTMANAGER_BEARER_SECRET", "")
BASIC_USERNAME = os.environ.get("ALERTMANAGER_USERNAME", "")
BASIC_PASSWORD = os.environ.get("ALERTMANAGER_PASSWORD", "")

def verify_authorization(header) -> bool:
    if not header:
        return False
    scheme, _, value = header.partition(" ")
    if scheme.lower() == "bearer":
        if not BEARER_SECRET:
            return False
        return hmac.compare_digest(value.encode("utf-8"), BEARER_SECRET.encode("utf-8"))
    if scheme.lower() != "basic" or not BASIC_USERNAME or not BASIC_PASSWORD:
        return False
    try:
        decoded = base64.b64decode(value.strip(), validate=True)
    except (binascii.Error, ValueError):
        return False
    user, sep, password = decoded.partition(b":")
    if not sep:
        return False
    user_ok = hmac.compare_digest(user, BASIC_USERNAME.encode("utf-8"))
    pass_ok = hmac.compare_digest(password, BASIC_PASSWORD.encode("utf-8"))
    return user_ok and pass_ok

```

Either credential is static. It authenticates the sender, not the body, so a modified payload still passes, and with no timestamp a captured request can be replayed. Serve the endpoint over HTTPS only, keep the credential in a file rather than in `alertmanager.yml`, and rotate it in Alertmanager and in your receiver together.

> **Make Alertmanager webhooks production-ready.** [Hookdeck Event Gateway](/event-gateway) checks the Basic or Bearer credentials you configure, queues every alert notification durably, and retries delivery to your service on a schedule you control.

## Prometheus Alertmanager webhook limitations and pain points

### The same alert, again and again

**The Problem:** Your receiver sees the same firing alert every few hours, or a flurry of notifications for one group.

**Why It Happens:** Alertmanager notifies per group, not per alert. After `group_wait` (30 seconds by default) it sends the first notification, then checks every `group_interval` (5 minutes) for changes, and repeats an unchanged notification after `repeat_interval` (4 hours). Each notification carries the whole group, so an alert you have already seen is sent again whenever the group changes.

**Workarounds:**

* Key your state on each alert's `fingerprint` and `status`, and upsert rather than insert.
* Use `groupKey` to identify the group, which the docs suggest for deduplication.
* Tune `group_by`, `group_interval` and `repeat_interval` per route rather than filtering in the receiver.

**How Hookdeck Can Help:** Hookdeck [deduplication](/docs/deduplication) can drop repeated notifications before they reach your service, based on fields you choose, such as `groupKey`, `status` and the alert fingerprints.

### Undocumented retries and no default timeout

**The Problem:** A slow or failing receiver delays notifications, and you cannot tell from the docs how long Alertmanager keeps trying.

**Why It Happens:** `timeout` defaults to `0s`, meaning no timeout, and setting it above `group_interval` has no effect. `group_interval` also bounds each send, so a notification that takes longer than that is cancelled. The docs say a timed-out request can be retried but do not document the schedule or which status codes count as success.

**Workarounds:**

* Set `timeout` to a few seconds below your `group_interval`.
* Respond with a `2xx` as soon as the credentials are checked and the notification is stored.

**How Hookdeck Can Help:** Hookdeck accepts each notification as soon as it arrives and stores it, then delivers to your service with [retries](/docs/retries) you configure. [Issues](/docs/issues) alert you when deliveries to your service start failing, which matters most when the thing failing is your alert path.

### Large groups get cut off

**The Problem:** A big incident produces a notification so large your receiver rejects it, or one that is missing alerts.

**Why It Happens:** By default every alert in the group is included. With `max_alerts` set, the extra alerts are dropped from the message and only counted in `truncatedAlerts`.

**Workarounds:**

* Set `max_alerts` to a size your receiver handles, and treat a non-zero `truncatedAlerts` as a signal to follow `externalURL` back to Alertmanager for the full list.
* Group more narrowly with `group_by` so groups stay small.

**How Hookdeck Can Help:** Hookdeck stores every notification with its full body, so you can inspect exactly what a large group sent and replay it after fixing your handler.

### Only two of the auth methods are checked

**The Problem:** A receiver configured with `oauth2` or a custom header never verifies in Hookdeck's Prometheus Alertmanager source.

**Why It Happens:** The source checks the `Authorization` header from `basic_auth` or from `authorization` with the `Bearer` type. An OAuth 2.0 token changes on every refresh, and custom headers are not part of this source type.

**Workarounds:**

* Use `basic_auth` or `authorization` on receivers that point at Hookdeck.
* If you are tied to another method, check it in your own receiver instead.

**How Hookdeck Can Help:** With a supported method, the source compares the credentials you enter. Hookdeck answers Alertmanager with a `200` either way, but requests without matching credentials are recorded as failed verification and never reach your service.

## Best practices

### Process each alert, not the group

Loop over `alerts` and act on each one's `status`, `labels` and `fingerprint`. The group-level `status` hides resolved alerts inside a firing group.

### Keep `send_resolved` on

Resolved notifications let your receiver close incidents automatically. Without them, you need another way to learn that an alert has stopped.

### Keep secrets in files

Use `password_file` or `credentials_file` so the credential never appears in `alertmanager.yml` or in version control.

### Avoid custom payloads unless you need them

The version 4 body is documented and stable. A `payload` template is unvalidated and unsupported, so prefer reshaping the body after it arrives.

### Acknowledge fast and process asynchronously

Check the credentials, store the notification, return `200`, and do the work afterwards. See [why to process webhooks asynchronously](/webhooks/guides/why-implement-asynchronous-processing-webhooks).

## Conclusion

Alertmanager's webhook receiver sends one version 4 JSON notification per alert group, with no signature and no event type, authenticated by `basic_auth` or a Bearer `authorization` header from `http_config`. Keep the credential in a file, compare it in constant time over HTTPS, process each alert by its own `status` and `fingerprint`, expect repeats from grouping and `repeat_interval`, and set a `timeout` so a slow receiver cannot hold notifications up.

[Hookdeck Event Gateway](https://hookdeck.com) checks the credentials, keeps every notification, and retries and replays delivery to your service, so a deploy on your side does not cost you an alert.

[Get started with Hookdeck](https://dashboard.hookdeck.com/signup) for free and handle Prometheus Alertmanager webhooks reliably in minutes.