# PagerDuty Webhooks

PagerDuty sends **outbound V3 webhooks** when incidents and services change —
triggered, acknowledged, escalated, reassigned, resolved, annotated, and more.

> **This skill targets V3 webhook subscriptions** — the current and only
> supported generation, created via
> `POST https://api.pagerduty.com/webhook_subscriptions`.
> 
> Canonical docs: [Webhooks Overview](https://docs.pagerduty.com/developer/webhooks-overview),
> [Verifying Signatures](https://docs.pagerduty.com/developer/verifying-webhook-signatures),
> [Behaviour](https://docs.pagerduty.com/developer/webhook-behavior).

## What This Is Not

* **V1 webhook extensions** — not covered, EOL October 2022 (they no longer function).
* **V2 webhook extensions** — legacy. End-of-support 31 Oct 2022, still
  functioning (no EOL date set) but receiving no fixes or features. Their
  payload is a `messages[]` array with event strings like `incident.trigger`
  (singular, no `d`) — deliberately different from V3's `incident.triggered`.
  V2 extensions are **not** signed with `X-PagerDuty-Signature`, so none of the
  verification here applies to them. [Migrate to V3](https://docs.pagerduty.com/integrations/webhooks#migration-guide).
* **PagerDuty Events API v1/v2** (`events.pagerduty.com`) — the opposite
  direction. You *send* alerts and change events *to* PagerDuty. Not webhooks.
* **Custom Incident Actions / "Generic Webhooks"** — same delivery pipeline,
  but a 16-second response timeout instead of 5 (see [Delivery](#delivery-semantics)).
* **Lookalikes:** PagerTree, Pagerly, Opsgenie, incident.io. Unrelated companies.

## When to Use This Skill

* How do I receive PagerDuty webhooks?
* How do I verify a PagerDuty webhook signature?
* Why is my `X-PagerDuty-Signature` verification failing?
* Why does `X-PagerDuty-Signature` contain two signatures?
* How do I handle `incident.triggered`, `incident.acknowledged` and `incident.resolved`?
* Where do I get the PagerDuty webhook signing secret?
* Does PagerDuty send a handshake or validation request? (No.)
* How do I de-duplicate PagerDuty webhook retries?

## Verification (core)

HMAC-SHA256 over the **raw request body**, lowercase hex (Base16), in
`X-PagerDuty-Signature`. The header may carry **multiple** comma-separated
signatures — accept if **any** matches. Keyed with the subscription secret.

```javascript
const crypto = require('crypto');

// X-PagerDuty-Signature: v1=<hex>,v1=<hex>   <- multiple = secret rotation
function verifyPagerDutySignature(rawBody, signatureHeader, secret) {
  if (!signatureHeader || !secret) return false;          // fail closed
  // HMAC over the RAW bytes. Raw digest, not hex — compare decoded bytes.
  const expected = crypto.createHmac('sha256', secret).update(rawBody).digest();
  return signatureHeader.split(',').some((entry) => {
    const part = entry.trim();
    if (!part.startsWith('v1=')) return false;  // IGNORE unknown versions, don't fail
    const candidate = Buffer.from(part.slice(3), 'hex'); // bad hex -> wrong length
    return (
      candidate.length === expected.length &&
      crypto.timingSafeEqual(candidate, expected)   // length guard FIRST: it throws
    );
  });
}

```

```python
import hashlib
import hmac

SIGNATURE_PREFIX = "v1="  # the current and only signature version

def verify_pagerduty_signature(raw_body: bytes, signature_header, secret) -> bool:
    if not signature_header or not secret:
        return False                      # fail closed
    # HMAC-SHA256 over the RAW body bytes, lowercase hex (Base16) — not base64.
    expected = hmac.new(secret.encode("utf-8"), raw_body, hashlib.sha256).hexdigest()
    expected_bytes = expected.encode("ascii")
    matched = False
    for entry in signature_header.split(","):
        part = entry.strip()              # defensive: PagerDuty sends no space
        if not part.startswith(SIGNATURE_PREFIX):
            continue                      # IGNORE unknown versions, don't fail
        candidate = part[len(SIGNATURE_PREFIX):].lower()
        # compare_digest on BYTES: two str args raise TypeError on non-ASCII,
        # and a forged header can carry anything. No early break — the work
        # stays independent of which entry matched.
        if hmac.compare_digest(candidate.encode("utf-8"), expected_bytes):
            matched = True
    return matched

```

> **For complete handlers with tests**, see [examples/express/](https://github.com/hookdeck/webhook-skills/tree/main/skills/pagerduty-webhooks/examples/express/), [examples/nextjs/](https://github.com/hookdeck/webhook-skills/tree/main/skills/pagerduty-webhooks/examples/nextjs/), [examples/fastapi/](https://github.com/hookdeck/webhook-skills/tree/main/skills/pagerduty-webhooks/examples/fastapi/).

**There is no timestamp and no nonce in the signed content**, so there is no
replay window to check. Do not add a tolerance or stale-time check — you would
be inventing a field that does not exist.

### Why manual HMAC and not an SDK

PagerDuty's JavaScript client (`@pagerduty/pdjs`) and Python client (`pdpyras`)
are **REST API clients with no webhook-verification helper** — there is nothing
to call. The only official verifier is in the Go client,
[`webhookv3/webhookv3.go`](https://github.com/PagerDuty/go-pagerduty/blob/master/webhookv3/webhookv3.go),
which is the authoritative reference for exact behaviour and is what the
examples here mirror. Use `node:crypto` / Python `hmac` + `hashlib` directly.

## Gotchas That Actually Bite

**Use the raw body.** PagerDuty, verbatim: *"Verifying PagerDuty webhook
signatures requires the unaltered raw body of the request sent to you. Ensure
that any frameworks or middleware you are using have not manipulated or
formatted the request body."* So `express.raw({ type: 'application/json' })`,
`await request.text()` in Next.js, `await request.body()` in FastAPI. Never
re-serialise parsed JSON.

**UTF-8, not latin-1.** PagerDuty: *"PagerDuty webhook payloads support unicode
characters. If your implementation is converting the request body from string to
bytes [or vice-versa], ensure that you are using the proper UTF-8 character
encoding."* Incident titles routinely contain non-ASCII.

**The header can hold more than one signature.** Verbatim from the docs:

```
X-PagerDuty-Signature: v1=f03de6f61df6e454f3620c4d6aca17ad072d3f8bbb2760eac3b2ad391b5e8073,v1=130dcacb53a94d983a37cf2acba98e805a1c37185309ba56fdcccbcf00d6dd8b

```

(The docs render that across lines for readability and note *"the actual header
value is sent as a single string without any new lines"*.) During a secret
rotation the same body is signed once per active secret and the results are
concatenated. **A verifier that compares only the whole header, or only the
first entry, breaks mid-rotation.** PagerDuty emits no space after the commas;
trimming each element defensively is harmless, but don't depend on a space.

**Ignore unknown signature versions; don't fail on them.** `v1` is the current
and only version. Skipping non-`v1=` entries is how a future `v2=` rolls out
without breaking your receiver.

**Non-hex candidates are skipped, not fatal.** Mirrors the Go client, which
hex-decodes each candidate and `continue`s past anything undecodable.

**`crypto.timingSafeEqual` throws on length mismatch.** Guard lengths first. An
uncaught throw becomes a 500, which PagerDuty retries for 48 hours.

**Python: `hmac.compare_digest` on BYTES.** Two `str` arguments raise
`TypeError` on non-ASCII input, and a forged header can carry anything.

**Status codes matter for retries.** Any 4xx except 429 is **permanent** — no
retry. 5xx, 429 and timeouts are retried for up to 48 hours. So reject forged
requests with a 4xx (400 for a missing/malformed header or empty body, 403 for a
signature mismatch), and never return 5xx for "bad signature" or PagerDuty will
hammer you for two days.

**Fail closed when the secret is unset.** 500 with a clear message (or refuse to
boot). Never skip verification because `PAGERDUTY_WEBHOOK_SECRET` is missing.

**There is no handshake.** No challenge, no echo, no `X-Hook-Secret` exchange,
no subscription-confirmation POST. The secret arrives in the create-subscription
**API response**, not over the wire. Don't build an endpoint for one. The one
delivery you *can* ask for is an explicit test: `POST /webhook_subscriptions/{id}/ping` sends a `pagey.ping` event (signing is not
documented for it — see Testing).

**`event.agent` and `event.client` can be `null`.** A `null` agent often means
automation rather than a person. `event.agent.id` will throw on
`service.updated`-style events. The documented `service.updated` example has
both `null`.

**`incident.service_updated` uses an underscore** — it is *not*
`incident.service.updated`, and it is a different event from `service.updated`.

## Envelope

A V3 payload contains exactly **one** `event` object by design.

| Field | Type | Notes |
| --- | --- | --- |
| `event.id` | String | Unique event id. Usable for de-duplication. |
| `event.event_type` | String | e.g. `incident.priority_updated`. **Route on this.** |
| `event.resource_type` | String | Root resource — currently `incident`, `service` or `scribe`. Can differ from `data.type`. |
| `event.occurred_at` | DateTime | ISO 8601. |
| `event.agent` | Object or `null` | [Resource Reference](https://docs.pagerduty.com/developer/resource-references) for who/what initiated it. `null` often means automation. |
| `event.client` | Object or `null` | e.g. `{"name": "PagerDuty"}`. |
| `event.data` | Object | Type-specific payload. Carries its own `type` discriminator. |

```json
{
  "event": {
    "id": "5ac64822-4adc-4fda-ade0-410becf0de4f",
    "event_type": "incident.priority_updated",
    "resource_type": "incident",
    "occurred_at": "2020-10-02T18:45:22.169Z",
    "agent": {
      "html_url": "https://acme.pagerduty.com/users/PLH1HKV",
      "id": "PLH1HKV",
      "self": "https://api.pagerduty.com/users/PLH1HKV",
      "summary": "Tenex Engineer",
      "type": "user_reference"
    },
    "client": { "name": "PagerDuty" },
    "data": {
      "id": "PGR0VU2",
      "type": "incident",
      "self": "https://api.pagerduty.com/incidents/PGR0VU2",
      "html_url": "https://acme.pagerduty.com/incidents/PGR0VU2",
      "number": 2,
      "status": "triggered",
      "incident_key": "d3640fbd41094207a1c11e58e46b1662",
      "created_at": "2020-04-09T15:16:27Z",
      "title": "A little bump in the road",
      "service": { "id": "PF9KMXH", "summary": "API Service", "type": "service_reference" },
      "assignees": [{ "id": "PTUXL6G", "summary": "User 123", "type": "user_reference" }],
      "priority": { "id": "PSO75BM", "summary": "P1", "type": "priority_reference" },
      "urgency": "high",
      "resolve_reason": null
    }
  }
}

```

Route on `event.event_type`; use `event.data.type` to pick the data schema.
`data.priority` can be `null` when no priority is set. Full field lists and
every `event.data` shape are in [references/overview.md](https://github.com/hookdeck/webhook-skills/blob/main/skills/pagerduty-webhooks/references/overview.md).

## Event Types

The complete V3 list, with the `data.type` each carries. PagerDuty: *"Additional
event types may be added to this list over time"*, and it may also ship
[Early Access events](https://docs.pagerduty.com/developer/early-access-webhooks)
without notice — **so your handler needs a default branch and must not throw on
an unrecognised `event_type`.**

| Event type | `data.type` | Sent when |
| --- | --- | --- |
| `incident.triggered` | `incident` | Incident newly created/triggered |
| `incident.acknowledged` | `incident` | Incident acknowledged |
| `incident.unacknowledged` | `incident` | Incident unacknowledged |
| `incident.resolved` | `incident` | Incident resolved |
| `incident.reopened` | `incident` | Incident reopened |
| `incident.escalated` | `incident` | Escalated to another user in the **same** escalation level |
| `incident.delegated` | `incident` | Reassigned to another **escalation policy** |
| `incident.reassigned` | `incident` | Reassigned to another **user** |
| `incident.priority_updated` | `incident` | Priority changed |
| `incident.service_updated` | `incident` | The incident's service changed (**underscore**) |
| `incident.incident_type.changed` | `incident` | Incident type changed |
| `incident.annotated` | `incident_note` | A note was added (**not** `incident.note.created`) |
| `incident.conference_bridge.updated` | `incident_conference_bridge` | Conference number and/or URL updated |
| `incident.custom_field_values.updated` | `incident_field_values` | Custom field values updated |
| `incident.status_update_published` | `incident_status_update` | A status update was added |
| `incident.responder.added` | `incident_responder` | A responder was added |
| `incident.responder.replied` | `incident_responder` | A responder replied to a request |
| `incident.role.assigned` | `incident_role_assignment` | A role was assigned **or unassigned** |
| `incident.task.created` | `incident_task` | Task created |
| `incident.task.updated` | `incident_task` | Task updated |
| `incident.task.completed` | `incident_task` | Task completed |
| `incident.action_invocation.created` | `incident_action_invocation` | Action invocation created |
| `incident.action_invocation.updated` | `incident_action_invocation` | Action invocation updated |
| `incident.action_invocation.terminated` | `incident_action_invocation` | Action invocation terminated |
| `incident.workflow.started` | `incident_workflow_instance` | Incident workflow started |
| `incident.workflow.completed` | `incident_workflow_instance` | Incident workflow completed |
| `scribe.summary.created` | `scribe_summary` | Scribe Agent generated (or failed to generate) a meeting summary. **Only for `account_reference` subscriptions** |
| `service.created` | `service` | Service created |
| `service.updated` | `service` | Service updated |
| `service.deleted` | `service` | Service deleted |
| `service.custom_field_values.updated` | `service_field_values` | Service custom field values updated |

Plus one that is **not subscribable** and not in that table: `pagey.ping`, which
PagerDuty delivers — signed, with a `resource_type` and `data` shape unlike any
documented event — when someone calls
`POST /webhook_subscriptions/{id}/ping`. It must fall through your default
branch.

Scoped-OAuth scopes: `incidents.read` for every `incident.*` **except**
`incident.workflow.*`, which needs `incident_workflows.read`; `services.read`
for `service.*`; `pd_advance.write` for `scribe.summary.created`.

## Delivery Semantics

* **POST**, `Content-Type: application/json`, **one event per request**. No batching.
* **Respond 2xx within 5 seconds** (16 seconds for webhooks generated from
  Custom Incident Actions). PagerDuty recommends returning **`202 Accepted`**
  immediately and processing asynchronously — the examples here verify, enqueue,
  then respond 202.
* **Retries for up to 48 hours** on: no response/timeout, 5xx, 429, connection
  failure, expired TLS certificate, DNS failure. **No retry** on any other 4xx,
  other TLS errors, or a 401 after a successful OAuth refresh.
* **Head-of-line blocking:** while a webhook is being retried, subsequent
  webhooks for the same subscription **and resource id** are queued.
* **Temporary disablement:** after **3 consecutive dropped** webhooks the
  subscription is disabled for **24 hours** and its queued webhooks are dropped.
  Re-enable from the webhooks dashboard ("Needs Attention") or the
  "Enable a webhook subscription" REST endpoint.
* **Ordering** is guaranteed per subscription + incident, in generation order.
* **At-least-once delivery.** De-duplicate on the **`X-Webhook-Id`** header:
  unique per webhook, **repeated across delivery attempts** of that webhook.
  (`event.id` works too; `X-Webhook-Id` is the documented one.) There is no
  documented `X-PagerDuty-Event` header, no delivery-timestamp header and no
  documented V3 User-Agent — don't key on headers that aren't documented.
* **Size limit:** delivery and ordering guaranteed up to **55 KB (56320 bytes)**.
  Above that PagerDuty tries to omit event details — the affected fields are on
  the **first `log_entry`'s `channel` object** (`details`,
  `cef_details.details`, `body`), replaced with an omission message and with
  `details_omitted` / `cef_details.details_omitted` / `body_omitted` flipped
  `false` → `true`. 55 KB–256 KB is best-effort (may be dropped or out of
  order). **Over 256 KB is always dropped.** If you cap request body size, the
  cap must be at least 256 KB.
* **Regions:** US (`api.pagerduty.com`) and EU (`api.eu.pagerduty.com`). Same
  signing scheme in both.
* Any publicly reachable host and port, http or https (https strongly
  preferred); a custom port is appended as `:port`.

## Other Security Layers

Only the signature protects **payload integrity**. These are defence in depth —
don't confuse them with verification.

* **Mutual TLS (PagerDuty's own recommendation).** PagerDuty presents a client
  certificate on request. Trust the **DigiCert Global Root G2**, set verify
  depth **2** (its leaf is signed by the intermediate "DigiCert Global G2 TLS
  RSA SHA256 2020 CA1"), and check the client cert Subject CN is
  `webhooks.pagerduty.com` (US) or `webhooks.eu.pagerduty.com` (EU). Client
  certs rotate **yearly** — pin the **root**, not the leaf. PagerDuty also
  verifies *your* server cert: it must chain to a CA in Mozilla's included-CA
  list (self-signed is dropped), the chain must be presented **in order**, and
  PagerDuty's delivery system supports **TLS v1.2 only**. This is server config,
  not app code — nginx/Apache snippets in
  [references/verification.md](https://github.com/hookdeck/webhook-skills/blob/main/skills/pagerduty-webhooks/references/verification.md).
* **OAuth 2.0 client credentials.** A subscription can be associated with an
  OAuth client so deliveries carry a bearer token.
* **IP safelists.** PagerDuty publishes per-region lists, shared across all
  customers and *subject to change* — fetch them at runtime rather than
  hardcoding:
  [US](https://docs.pagerduty.com/ip-safelists/webhooks-us-service-region)
  ([JSON](https://docs.pagerduty.com/ip-safelists/webhooks-us-service-region-json)),
  [EU](https://docs.pagerduty.com/ip-safelists/webhooks-eu-service-region)
  ([JSON](https://docs.pagerduty.com/ip-safelists/webhooks-eu-service-region-json)).
  These are the webhook + workflow-action egress IPs and are **different from
  the REST API IPs** ([/developer/rest-api-ips](https://docs.pagerduty.com/developer/rest-api-ips)).
* **Basic auth in the URL** (`https://user:pass@host`) is supported; special
  characters must be percent-encoded. Mentioned for completeness, not recommended.
* **`custom_headers`** on the subscription are delivered **verbatim** to your
  endpoint (they are redacted in GET API responses, but not on delivery). A
  shared-secret header is possible, but it is **not** a substitute for the
  signature.

## Environment Variables

```bash
# REQUIRED. The webhook subscription's signing secret, returned as
# delivery_method.secret in the POST /webhook_subscriptions response.
# Shown at creation time only. Used AS-IS as UTF-8 HMAC key bytes.
# NOT an API key / REST token, NOT an Events API routing key.
PAGERDUTY_WEBHOOK_SECRET=

```

The examples **fail closed**: with `PAGERDUTY_WEBHOOK_SECRET` unset they reject
every delivery with a clear error rather than silently skipping verification.

## Setup

```bash
curl -X POST https://api.pagerduty.com/webhook_subscriptions \
  -H 'Authorization: Token token=YOUR_API_TOKEN' \
  -H 'Content-Type: application/json' \
  -d '{
    "webhook_subscription": {
      "type": "webhook_subscription",
      "delivery_method": {
        "type": "http_delivery_method",
        "url": "https://example.com/webhooks/pagerduty"
      },
      "description": "Incident webhooks",
      "events": ["incident.triggered", "incident.acknowledged", "incident.resolved"],
      "filter": { "type": "service_reference", "id": "P393ZNQ" }
    }
  }'

```

Capture `delivery_method.secret` from the response — **that is the signing key**.
Filters are `service_reference`, `team_reference` or `account_reference`; incident
events are scoped to incidents belonging to the filtered object. Full walkthrough
in [references/setup.md](https://github.com/hookdeck/webhook-skills/blob/main/skills/pagerduty-webhooks/references/setup.md).

## Local Development

```bash
npx hookdeck-cli listen 3000 pagerduty --path /webhooks/pagerduty

```

No account required — the CLI creates a guest account on first run and gives you
a public HTTPS URL plus a web UI for inspecting requests (raw body and
`X-PagerDuty-Signature` included, which is what you want when debugging). Use
the printed URL as the subscription's `delivery_method.url`. (Use `8000` for the
FastAPI example.)

Then fire a signed test delivery without waiting for a real incident:

```bash
curl -X POST https://api.pagerduty.com/webhook_subscriptions/PWHSUB1/ping \
  -H 'Authorization: Token token=YOUR_API_TOKEN'

```

PagerDuty returns `202` and delivers a **`pagey.ping`** event (needs the
`webhook_subscriptions.write` scope). It goes through the subscription's normal
delivery method; PagerDuty does not document whether the ping is signed, so
check your own logs rather than assuming a signed ping either way. PagerDuty sends no handshake, challenge or
validation request — that ping is the only unsolicited delivery you can trigger.
`pagey.ping` is not subscribable, so it lands in your default branch.

## Reference Materials

* [references/overview.md](https://github.com/hookdeck/webhook-skills/blob/main/skills/pagerduty-webhooks/references/overview.md) — Envelope, all 31 event types, every `event.data` shape, delivery semantics, idempotency
* [references/setup.md](https://github.com/hookdeck/webhook-skills/blob/main/skills/pagerduty-webhooks/references/setup.md) — Creating a subscription, capturing the secret, filters, custom headers, re-enabling a disabled subscription, secret rotation
* [references/verification.md](https://github.com/hookdeck/webhook-skills/blob/main/skills/pagerduty-webhooks/references/verification.md) — `X-PagerDuty-Signature` byte by byte, multi-signature rotation, mutual TLS config, OAuth, IP safelists, debugging failures