# GitHub webhooks in production: duplicates, ordering, retries and timeouts

Agent skill: `npx skills add hookdeck/webhook-skills --skill github-webhooks` ([details](https://hookdeck.com/webhooks/github/skill))

This guide covers the problems GitHub webhooks cause in production and how to handle each one. What GitHub sends and how it's signed is on the [GitHub webhooks](/webhooks/github) overview, every event type is on [GitHub webhook events](/webhooks/github/events), and signature verification code is on [Verify GitHub webhooks](/webhooks/github/verify).

## At a glance

| Problem | What to do | With Event Gateway |
| --- | --- | --- |
| [Timeouts](#timeouts) | Return a 2xx within 10 seconds; process in a background job | Accepts the delivery at once, then queues it for your service |
| [Duplicates](#duplicates) | Record each `X-GitHub-Delivery` value; make handlers idempotent | Drops repeats before they reach your handler |
| [Ordering](#ordering) | Compare timestamps in the payload; fetch current state when in doubt | Logs every event and attempt, so you can see arrival order when debugging |
| [Retries](#retries) | Redeliver failures within 3 days, with a scheduled job | Accepts deliveries while your service is down and retries them |
| [Choosing events](#choosing-events) | Subscribe only to the events you process; check the event and the action | Filters and routes events by header or body |
| [Testing](#testing) | Forward events to localhost with `gh webhook forward` or a proxy URL | Forwards to localhost and replays past events |
| [Other limits](#other-limits) | Plan for the 25 MB cap, 20 webhooks per event type, skipped bulk pushes and throttling | Fans one webhook out to many destinations at a rate you set |

## GitHub webhook timeouts and slow handlers {#timeouts}

Return a 2xx status within 10 seconds, then do the work in a background job.

**Why it happens:** If your server takes longer than 10 seconds to respond, GitHub closes the connection and records the delivery as a failure. GitHub doesn't retry it, so a slow handler loses the event unless something redelivers it. Handlers that call other APIs or run long jobs before responding go over the limit, and a traffic spike slows them further until a backlog builds.

**Fix it in your handler:**

1. Verify the signature and return a 2xx straight away.
2. Push the event onto a queue, such as RabbitMQ, Apache Kafka or Resque, and process it in a worker. See [why to process webhooks asynchronously](/webhooks/guides/why-implement-asynchronous-processing-webhooks).
3. Rate limit the workers, so a burst of events can't exhaust your API server's memory and take it down.

**With Event Gateway:** Hookdeck accepts each GitHub delivery immediately, then queues events and delivers them to your service at a rate you set, with its own retries. A slow handler no longer fails a GitHub delivery.

## Duplicate GitHub webhooks {#duplicates}

Record each `X-GitHub-Delivery` value you process and skip repeats.

**Why it happens:** GitHub doesn't redeliver failures by itself, so repeats come from redeliveries: someone clicks Redeliver, or a script calls the REST API. A redelivery has the same `X-GitHub-Delivery` value as the original. Your handler may also have done the work and then missed the 10-second limit, so GitHub records a failure and your redelivery job sends an event you've already processed.

Separate events can describe one change too. Deleting a branch produces a `delete` event and a `push` event, and deduplicating on the delivery ID won't connect them.

**Fix it in your handler:**

1. Track processed delivery IDs, using a database table with a unique constraint on the ID.
2. Make every handler [idempotent](/webhooks/guides/implement-webhook-idempotency): the same result however many times it runs.
3. Wrap the side effect and the ID record in one transaction.
4. Subscribe to `push` or to `create` and `delete`, not both, unless you handle the overlap.

**With Event Gateway:** Hookdeck [deduplicates](/docs/deduplication) events before they reach your endpoint, so repeats are filtered at the infrastructure level rather than in each handler.

## Out-of-order GitHub webhooks {#ordering}

GitHub may deliver webhooks in a different order from the events that caused them. Compare the timestamps in the payload when order matters.

**Why it happens:** GitHub's docs say delivery order can differ from the order events took place, and deliveries can take a few minutes to arrive. A redelivery also lands well after newer events. A `closed` action can reach you before the `edited` action that came first. For more on the general problem, see [Webhook Ordering: Why It's Hard and How to Handle It](/webhooks/guides/webhook-ordering-why-its-hard-and-how-to-handle-it).

**Fix it in your handler:**

1. Use the payload's timestamps, such as `updated_at` on an issue or pull request, and don't let an older event overwrite newer state.
2. Allow for ties. Timestamps in GitHub's example payloads are recorded in whole seconds, so fetch the object's current state from the REST API when two events are too close to call.
3. Write state transitions that tolerate events in any order.

**With Event Gateway:** Hookdeck logs every event and delivery attempt, so when state looks wrong you can see the order events actually arrived in and replay them.

## GitHub webhook retries and redelivery {#retries}

GitHub doesn't retry failed deliveries. Redeliver them yourself within 3 days, or run a job that does.

**Why it happens:** GitHub records a delivery as failed if your server is down, takes longer than 10 seconds or returns a 4xx or 5xx status, and then moves on. You can redeliver anything from the past 3 days. GitHub's webhook docs don't describe any alert when deliveries fail, or disabling a webhook because of repeated failures, so watch for failures yourself.

**Fix it in your handler:**

1. Return a 2xx for events you accept or ignore, and an accurate error otherwise, because every 4xx and 5xx counts as a failure.
2. Check the webhook's Recent deliveries list, or list deliveries through the REST API. A failed delivery has a `status` other than `OK`.
3. Run a [scheduled redelivery job](https://docs.github.com/en/webhooks/using-webhooks/handling-failed-webhook-deliveries) that lists deliveries since its last run and redelivers the failures. GitHub publishes example scripts for repository, organization and app webhooks. Marketplace and Sponsors webhooks have no deliveries API, so a job can't cover them.
4. After an outage longer than 3 days, reconcile from the REST API, because those deliveries can no longer be redelivered.

For the errors GitHub records against a delivery, see [troubleshooting GitHub webhooks](/webhooks/platforms/guide-troubleshooting-github-webhooks).

**With Event Gateway:** Hookdeck captures every GitHub delivery while your services are down, so GitHub's deliveries succeed. Failed events can be [retried](/docs/retries) automatically, manually or in bulk once you recover.

## Choosing which GitHub events to listen to {#choosing-events}

Subscribe each webhook only to the events your integration processes, then check both the event and the action before you act.

**Why it matters:** Every extra event type is a request your server has to accept, verify and discard, and more deliveries make throttling more likely. GitHub has dozens of event types and many have several actions, so a single `pull_request` subscription covers actions such as `opened`, `closed` and `labeled`. The full list, with sample payloads, is on [GitHub webhook events](/webhooks/github/events). For ideas on where to start, see [top use cases for GitHub webhooks](/webhooks/platforms/top-use-cases-github-webhooks).

**Fix it in your handler:**

1. List the events your code handles and subscribe to exactly those.
2. Branch on the `X-GitHub-Event` header and the body's `action` field, and ignore what you don't recognise. GitHub continues to add event types and actions.
3. Check availability: each event is only offered on certain webhook types, so an organization webhook can subscribe to `team` and a repository webhook can't. A GitHub App's events depend on its permissions.
4. Prefer the current event where GitHub names one: `package` over `registry_package`, and `dependabot_alert` over `repository_vulnerability_alert`, which is closing down.

**With Event Gateway:** Hookdeck [filters](/docs/filters) and routes events on headers or body, so you can route on the event and the action and let one GitHub webhook feed several services.

## Testing and debugging GitHub webhooks locally {#testing}

Forward real GitHub events to your local server with the GitHub CLI or a proxy URL rather than deploying to test.

**Why it happens:** GitHub rejects `localhost` and `127.0.0.1` as webhook URLs, so a local server needs something in between.

**Fix it in your handler:**

1. Use `gh webhook forward` from the `cli/gh-webhook` extension for repository and organization webhooks. Only one person at a time can forward each repository or organization, and it isn't supported in production.
2. For other webhook types, use a [smee.io](https://smee.io/) proxy URL. GitHub says never to use it in production, because its channels aren't authenticated.
3. Trigger the event for real, redeliver a past delivery, or use the REST API to send a `ping` or a test `push`.
4. Check your signature code against GitHub's published test values: the secret `It's a Secret to Everybody` and the payload `Hello, World!` should give `sha256=757107ea0eb2509fc211221cce984b8a37570b6d7586c22c46f4379c8b043e17`.
5. Keep fixtures of real payloads for edge cases; [GitHub webhook events](/webhooks/github/events) has sample payloads to start from.

**With Event Gateway:** The [Hookdeck CLI](/docs/cli) receives webhooks on your machine without a public server, and you can replay past events with one click. See [how to test and replay GitHub webhooks locally](/webhooks/platforms/how-to-test-and-replay-github-webhook-events-on-localhost-with-hookdeck).

## Other GitHub webhook limits {#other-limits}

Plan for GitHub's payload cap, webhook limit, skipped events and throttling before they reach production.

**25 MB payload cap.** GitHub doesn't deliver a payload above 25 MB, which can happen on a `create` event when many branches or tags are pushed at once. Monitor your payload sizes.

**Bulk pushes create no event.** GitHub creates no `push` event when more than 5,000 branches are pushed at once, or for tags when more than three are pushed at once. `create` and `delete` don't fire when more than three tags are created or deleted at once. Reconcile from the API after bulk operations.

**20 webhooks per event type.** A repository or organization can have up to 20 webhooks subscribed to each event type. GitHub suggests a proxy that forwards to more destinations. A GitHub App or Marketplace listing has only one webhook.

**Throttling and delays.** Deliveries can take a few minutes. During a surge GitHub may throttle deliveries to your account, and each affected delivery shows a `throttled_at` timestamp in the REST API.

**The form content type.** The REST API's `content_type` setting defaults to `form`, so a webhook created without it sends the JSON inside a `payload` form field, not as the body. Set `json` if your handler parses JSON.

**IP allow lists.** `GET /meta` lists the ranges GitHub delivers from, under `hooks`. GitHub changes them occasionally, so refresh your list on a schedule with a cron job. Webhooks don't support IPv6 yet.

**`sender` isn't always a person.** When GitHub can't resolve a user, `sender` is the `ghost` placeholder account, so don't rely on it in security or business logic.

**Preview events.** GitHub says webhook events for Discussions and for projects are in public preview and subject to change.

For a wider production checklist, see [taking control of your webhook reliability](/webhooks/guides/taking-control-of-your-webhook-reliability) and the [webhooks security checklist](/webhooks/guides/webhooks-security-checklist).

**With Event Gateway:** Hookdeck sends one GitHub webhook's events to as many destinations as you need, absorbs spikes in its queue, and delivers at a rate you configure.