GitLab webhooks in production: auto-disabling, duplicates, retries and timeouts

Published · Updated

This guide covers the problems GitLab webhooks cause in production and how to handle each one. What GitLab sends and how it's signed is on the GitLab webhooks overview, every event type is on GitLab webhook events, and signature verification code is on Verify GitLab webhooks.

At a glance

ProblemWhat to doWith Event Gateway
TimeoutsReturn a 200 or 201 within 10 seconds; process in a background jobAccepts the delivery at once, then queues it for your service
Retries and auto-disablingWatch for disabled webhooks; resend what you missedAccepts deliveries while your service is down, so GitLab's failure count doesn't grow
DuplicatesRecord each webhook-id; make handlers idempotentDrops repeats before they reach your handler
Choosing eventsSubscribe only to the events you process; return a 2xx for the restFilters and routes events by type
TestingSend test events; forward them to localhost with a CLIForwards to localhost and replays past events
Other limitsPlan for push limits, rate limits and short log retentionKeeps its own searchable log of every event

GitLab webhook timeouts and slow handlers

Return a 200 or 201 within 10 seconds, then do the work in a background job.

Why it happens: GitLab.com waits 10 seconds for a response. A timeout counts as a failure towards disabling the webhook, and GitLab warns that it can also lead to duplicate events. Handlers that query a database, call other APIs or start a build before responding go over the limit, and cold starts on serverless platforms use up part of the 10 seconds. On a self-managed instance, administrators can change the limit with gitlab_rails['webhook_timeout'].

Fix it in your handler:

  1. Verify the signature and return a 200 or 201 straight away. Consider an empty body, because GitLab stores response headers and bodies for later inspection.
  2. Push the event onto a queue (Redis, RabbitMQ or SQS) and process it in a worker. See why to process webhooks asynchronously.
  3. Keep the request path free of calls to other services.

For more on the causes, see how to solve GitLab webhook timeout errors.

With Event Gateway: Hookdeck accepts each GitLab delivery immediately, then queues events and delivers them to your service at a rate you set, with its own retries. Your handler can take as long as it needs without failing a GitLab delivery.

GitLab webhook retries and automatic disabling

GitLab's docs publish no retry schedule, and repeated failures disable the webhook. Watch for disabled webhooks and resend what you missed.

Why it happens: GitLab temporarily disables a project or group webhook after 4 consecutive failures. A failure is a 4xx or 5xx response, a timeout while connecting to your endpoint, or another HTTP error, and an invalid HTTP response counts too. A temporarily disabled webhook stays off for one minute at first, and the time extends with further failures up to 24 hours, after which GitLab re-enables it. After 40 consecutive failures the webhook is permanently disabled and GitLab won't re-enable it by itself. On GitLab Self-Managed, automatic disabling of project webhooks sits behind the auto_disabling_web_hooks feature flag, so check your instance.

GitLab's docs describe no queue for events while a webhook is disabled, so plan on those events not arriving. They mention retries only in passing: webhook-id is "consistent across webhook retries", and they warn of duplicates after a timeout. Treat any failed delivery as one you may have to recover yourself.

Fix it in your handler:

  1. Check the webhook list for the Temporarily disabled and Disabled badges, or read alert_status and disabled_until from the project webhooks API, and alert on them.
  2. Re-enable a disabled webhook by sending a test request. It's re-enabled if the test returns a 2xx.
  3. Resend failed deliveries from the webhook's Recent events section, which covers the last two days, or with POST /projects/:id/hooks/:hook_id/events/:hook_event_id/resend. The API lists events from the past 7 days and limits resends to five requests per minute. You can't resend once the webhook's URL has changed.
  4. After a longer outage, reconcile from the GitLab API, because those deliveries are out of reach.
  5. Keep deploys from failing deliveries. Four failed requests during a rolling deploy are enough to disable a webhook.

For recovery steps and workarounds, see how to solve GitLab's automatic webhook disabling.

With Event Gateway: Hookdeck accepts each GitLab delivery while your services are down, so GitLab sees a success and the failure count doesn't grow. Failed events can be retried automatically, manually or in bulk once you recover.

Duplicate GitLab webhooks

GitLab tells you to prepare for duplicate events if a webhook times out. Record each webhook-id you process and skip repeats.

Why it happens: A timeout can lead to a second delivery of an event you've already processed, and a resend repeats a delivery on purpose. Both keep the same ID: webhook-id is the same across retries, and it equals Idempotency-Key, the older header. webhook-id arrives from GitLab 19.0 and Idempotency-Key from 17.4.

Some repeats are separate deliveries that a delivery ID won't connect:

  • Identical webhooks on a project and on its group both fire for events in that project.
  • One action can trigger several event types, so a merge sends a merge request event and a push event.
  • Pipeline events fire on each status change, so one pipeline sends several.
  • A rejected deployment sends status: "rejected" and then status: "failed" for the same deployment_id.

Fix it in your handler:

  1. Track processed IDs, using a database table with a unique constraint on the ID.
  2. Make every handler idempotent: the same result however many times it runs.
  3. Wrap the side effect and the ID record in one transaction.
  4. Configure each webhook at one level, on the project or on the group.
  5. Correlate deployment events by deployment_id, and treat a failed that follows a rejected as one outcome.
def process_gitlab_webhook(headers, payload):
    delivery_id = headers.get('webhook-id') or headers.get('Idempotency-Key')

    # Skip anything already processed.
    if already_processed(delivery_id):
        return

    with db.transaction():
        do_processing(payload)
        mark_processed(delivery_id)

With Event Gateway: Hookdeck deduplicates events before they reach your endpoint, so repeats are filtered at the infrastructure level rather than in each handler.

Choosing which GitLab events to listen to

Subscribe each webhook only to the events your integration processes, and return a 2xx for anything you ignore.

Why it matters: Every failed response moves a webhook closer to being disabled. GitLab says to return a 400 or ignore the payload for an event you don't support, but any 4xx counts as a failure, so ignoring it is the safer choice. Extra events also count towards your namespace's rate limit on GitLab.com. The full list, with sample payloads, is on GitLab webhook events.

Fix it in your handler:

  1. List the events your code handles and select exactly those under Trigger.
  2. Branch on the X-Gitlab-Event header or the body's object_kind, and on object_attributes.action where the event has one. The header's value isn't always a restyling of the body's: a job event arrives as Job Hook with object_kind: build.
  3. Filter push events by branch with a wildcard pattern or an RE2 regular expression, in the webhook's settings.
  4. Select Confidential work items events if you need them, because a webhook with only Work item events selected doesn't receive them.
  5. Read the changes field on merge request events, because an event can fire with it empty.

With Event Gateway: Hookdeck filters and routes events on headers or body, so you can route on the event type and let one GitLab webhook feed several services.

Testing and debugging GitLab webhooks locally

Send test events from GitLab, and forward them to your local server through a tunnel rather than deploying to test.

Why it matters: GitLab.com can't reach a server on your machine, and GitLab blocks webhook requests to local network addresses by default, so a self-managed instance won't call 127.0.0.1 or a private range until an administrator allows it.

How to test:

  1. Send a test from the Test dropdown on the webhook list, choosing the event type. It needs at least one commit to test push events, and GitLab doesn't support testing for some event types. Test requests are rate limited to 5 each minute by default.
  2. Check the request and response under Recent events, where View details shows the body and headers of both.
  3. To inspect a payload without running anything, GitLab points to public tools such as Webhook.site, and warns you to use test tokens and rotate anything you send to them by accident.
  4. Keep fixtures of real payloads for edge cases; GitLab webhook events has sample payloads to start from.

With Event Gateway: The Hookdeck CLI receives webhooks on your machine without a public server, and you can replay past events with one click. See how to test and replay GitLab webhooks on localhost.

Other GitLab webhook limits

Plan for GitLab's push limits, rate limits, payload cap and short log retention before they reach production.

Large pushes send nothing. By default, a push that changes more than 3 branches or tags triggers no webhooks at all, with no notice. Self-managed administrators can raise push_event_hooks_limit; on GitLab.com, split bulk operations or reconcile from the API afterwards. A push of more than 20 commits lists only the newest 20 in commits.

Namespace rate limits. GitLab.com limits webhook calls per minute across a top-level namespace, from 500 on Free up to 13,000 on the largest Ultimate plans. When the limit is reached, all webhooks in the namespace are temporarily disabled and re-enabled in the next minute. Look for Webhook rate limit exceeded in your logs, and reduce the events that trigger webhooks or contact GitLab Support.

Short logs. Recent events shows the last two days, and the project webhooks API lists the past 7 days. Log deliveries on your side if you need longer. See webhook observability architecture if you're building this yourself.

Payload and webhook caps. A payload can reach 25 MB. GitLab.com allows 100 webhooks on a project and 50 on a group, not counting subgroups.

Recursive webhooks. GitLab blocks recursive webhooks, and a chain of webhooks that trigger other webhooks can make at most 100 requests to the instance. Check this if your handler calls the GitLab API. X-Gitlab-Event-UUID is shared across recursive webhooks.

Redacted and surprising fields. The email attribute is [REDACTED] for an author with no public email. On job events, commit.id is the pipeline's ID, not the commit's. Timestamps moved to ISO 8601 with millisecond precision in GitLab 19.0.

With Event Gateway: Hookdeck logs every event and delivery attempt, with the full request and response, so you can search past deliveries and replay them. It can't make GitLab send a webhook that GitLab's own limits suppress.

Keep an outage from disabling your GitLab webhook.

Four failures in a row disable a GitLab webhook. Hookdeck Event Gateway accepts every GitLab delivery, verifies the signature, queues events and retries to your service on your schedule.

Not affiliated with GitLab. Facts are compiled from public sources and linked inline.