GitHub webhooks in production: duplicates, ordering, retries and timeouts

Published · Updated

This guide covers the problems GitHub webhooks cause in production and how to handle each one. What GitHub sends and how it's signed is on the GitHub webhooks overview, every event type is on GitHub webhook events, and signature verification code is on Verify GitHub webhooks.

At a glance

ProblemWhat to doWith Event Gateway
TimeoutsReturn a 2xx within 10 seconds; process in a background jobAccepts the delivery at once, then queues it for your service
DuplicatesRecord each X-GitHub-Delivery value; make handlers idempotentDrops repeats before they reach your handler
OrderingCompare timestamps in the payload; fetch current state when in doubtLogs every event and attempt, so you can see arrival order when debugging
RetriesRedeliver failures within 3 days, with a scheduled jobAccepts deliveries while your service is down and retries them
Choosing eventsSubscribe only to the events you process; check the event and the actionFilters and routes events by header or body
TestingForward events to localhost with gh webhook forward or a proxy URLForwards to localhost and replays past events
Other limitsPlan for the 25 MB cap, 20 webhooks per event type, skipped bulk pushes and throttlingFans one webhook out to many destinations at a rate you set

GitHub webhook timeouts and slow handlers

Return a 2xx status within 10 seconds, then do the work in a background job.

Why it happens: If your server takes longer than 10 seconds to respond, GitHub closes the connection and records the delivery as a failure. GitHub doesn't retry it, so a slow handler loses the event unless something redelivers it. Handlers that call other APIs or run long jobs before responding go over the limit, and a traffic spike slows them further until a backlog builds.

Fix it in your handler:

  1. Verify the signature and return a 2xx straight away.
  2. Push the event onto a queue, such as RabbitMQ, Apache Kafka or Resque, and process it in a worker. See why to process webhooks asynchronously.
  3. Rate limit the workers, so a burst of events can't exhaust your API server's memory and take it down.

With Event Gateway: Hookdeck accepts each GitHub delivery immediately, then queues events and delivers them to your service at a rate you set, with its own retries. A slow handler no longer fails a GitHub delivery.

Duplicate GitHub webhooks

Record each X-GitHub-Delivery value you process and skip repeats.

Why it happens: GitHub doesn't redeliver failures by itself, so repeats come from redeliveries: someone clicks Redeliver, or a script calls the REST API. A redelivery has the same X-GitHub-Delivery value as the original. Your handler may also have done the work and then missed the 10-second limit, so GitHub records a failure and your redelivery job sends an event you've already processed.

Separate events can describe one change too. Deleting a branch produces a delete event and a push event, and deduplicating on the delivery ID won't connect them.

Fix it in your handler:

  1. Track processed delivery IDs, using a database table with a unique constraint on the ID.
  2. Make every handler idempotent: the same result however many times it runs.
  3. Wrap the side effect and the ID record in one transaction.
  4. Subscribe to push or to create and delete, not both, unless you handle the overlap.

With Event Gateway: Hookdeck deduplicates events before they reach your endpoint, so repeats are filtered at the infrastructure level rather than in each handler.

Out-of-order GitHub webhooks

GitHub may deliver webhooks in a different order from the events that caused them. Compare the timestamps in the payload when order matters.

Why it happens: GitHub's docs say delivery order can differ from the order events took place, and deliveries can take a few minutes to arrive. A redelivery also lands well after newer events. A closed action can reach you before the edited action that came first. For more on the general problem, see Webhook Ordering: Why It's Hard and How to Handle It.

Fix it in your handler:

  1. Use the payload's timestamps, such as updated_at on an issue or pull request, and don't let an older event overwrite newer state.
  2. Allow for ties. Timestamps in GitHub's example payloads are recorded in whole seconds, so fetch the object's current state from the REST API when two events are too close to call.
  3. Write state transitions that tolerate events in any order.

With Event Gateway: Hookdeck logs every event and delivery attempt, so when state looks wrong you can see the order events actually arrived in and replay them.

GitHub webhook retries and redelivery

GitHub doesn't retry failed deliveries. Redeliver them yourself within 3 days, or run a job that does.

Why it happens: GitHub records a delivery as failed if your server is down, takes longer than 10 seconds or returns a 4xx or 5xx status, and then moves on. You can redeliver anything from the past 3 days. GitHub's webhook docs don't describe any alert when deliveries fail, or disabling a webhook because of repeated failures, so watch for failures yourself.

Fix it in your handler:

  1. Return a 2xx for events you accept or ignore, and an accurate error otherwise, because every 4xx and 5xx counts as a failure.
  2. Check the webhook's Recent deliveries list, or list deliveries through the REST API. A failed delivery has a status other than OK.
  3. Run a scheduled redelivery job that lists deliveries since its last run and redelivers the failures. GitHub publishes example scripts for repository, organization and app webhooks. Marketplace and Sponsors webhooks have no deliveries API, so a job can't cover them.
  4. After an outage longer than 3 days, reconcile from the REST API, because those deliveries can no longer be redelivered.

For the errors GitHub records against a delivery, see troubleshooting GitHub webhooks.

With Event Gateway: Hookdeck captures every GitHub delivery while your services are down, so GitHub's deliveries succeed. Failed events can be retried automatically, manually or in bulk once you recover.

Choosing which GitHub events to listen to

Subscribe each webhook only to the events your integration processes, then check both the event and the action before you act.

Why it matters: Every extra event type is a request your server has to accept, verify and discard, and more deliveries make throttling more likely. GitHub has dozens of event types and many have several actions, so a single pull_request subscription covers actions such as opened, closed and labeled. The full list, with sample payloads, is on GitHub webhook events. For ideas on where to start, see top use cases for GitHub webhooks.

Fix it in your handler:

  1. List the events your code handles and subscribe to exactly those.
  2. Branch on the X-GitHub-Event header and the body's action field, and ignore what you don't recognise. GitHub continues to add event types and actions.
  3. Check availability: each event is only offered on certain webhook types, so an organization webhook can subscribe to team and a repository webhook can't. A GitHub App's events depend on its permissions.
  4. Prefer the current event where GitHub names one: package over registry_package, and dependabot_alert over repository_vulnerability_alert, which is closing down.

With Event Gateway: Hookdeck filters and routes events on headers or body, so you can route on the event and the action and let one GitHub webhook feed several services.

Testing and debugging GitHub webhooks locally

Forward real GitHub events to your local server with the GitHub CLI or a proxy URL rather than deploying to test.

Why it happens: GitHub rejects localhost and 127.0.0.1 as webhook URLs, so a local server needs something in between.

Fix it in your handler:

  1. Use gh webhook forward from the cli/gh-webhook extension for repository and organization webhooks. Only one person at a time can forward each repository or organization, and it isn't supported in production.
  2. For other webhook types, use a smee.io proxy URL. GitHub says never to use it in production, because its channels aren't authenticated.
  3. Trigger the event for real, redeliver a past delivery, or use the REST API to send a ping or a test push.
  4. Check your signature code against GitHub's published test values: the secret It's a Secret to Everybody and the payload Hello, World! should give sha256=757107ea0eb2509fc211221cce984b8a37570b6d7586c22c46f4379c8b043e17.
  5. Keep fixtures of real payloads for edge cases; GitHub webhook events has sample payloads to start from.

With Event Gateway: The Hookdeck CLI receives webhooks on your machine without a public server, and you can replay past events with one click. See how to test and replay GitHub webhooks locally.

Other GitHub webhook limits

Plan for GitHub's payload cap, webhook limit, skipped events and throttling before they reach production.

25 MB payload cap. GitHub doesn't deliver a payload above 25 MB, which can happen on a create event when many branches or tags are pushed at once. Monitor your payload sizes.

Bulk pushes create no event. GitHub creates no push event when more than 5,000 branches are pushed at once, or for tags when more than three are pushed at once. create and delete don't fire when more than three tags are created or deleted at once. Reconcile from the API after bulk operations.

20 webhooks per event type. A repository or organization can have up to 20 webhooks subscribed to each event type. GitHub suggests a proxy that forwards to more destinations. A GitHub App or Marketplace listing has only one webhook.

Throttling and delays. Deliveries can take a few minutes. During a surge GitHub may throttle deliveries to your account, and each affected delivery shows a throttled_at timestamp in the REST API.

The form content type. The REST API's content_type setting defaults to form, so a webhook created without it sends the JSON inside a payload form field, not as the body. Set json if your handler parses JSON.

IP allow lists. GET /meta lists the ranges GitHub delivers from, under hooks. GitHub changes them occasionally, so refresh your list on a schedule with a cron job. Webhooks don't support IPv6 yet.

sender isn't always a person. When GitHub can't resolve a user, sender is the ghost placeholder account, so don't rely on it in security or business logic.

Preview events. GitHub says webhook events for Discussions and for projects are in public preview and subject to change.

For a wider production checklist, see taking control of your webhook reliability and the webhooks security checklist.

With Event Gateway: Hookdeck sends one GitHub webhook's events to as many destinations as you need, absorbs spikes in its queue, and delivers at a rate you configure.

Stop losing GitHub events when your endpoint is down.

GitHub won't retry a failed delivery. Hookdeck Event Gateway accepts every GitHub delivery, verifies the signature, queues events and retries to your service on your schedule.

Not affiliated with GitHub. Facts are compiled from public sources and linked inline.