Zendesk webhooks in production: duplicates, ordering, retries and timeouts

Published · Updated

This guide covers the problems Zendesk webhooks cause in production and how to handle each one. What Zendesk sends and how it's signed is on the Zendesk webhooks overview, every event type is on Zendesk webhook events, and signature verification code is on Verify Zendesk webhooks.

Zendesk has two kinds of webhook, and some of the fixes differ. A webhook subscribed to Zendesk events always sends a POST with a fixed JSON body and a type field. A webhook connected to a trigger or automation sends the method, format and body you define, so it has no event type. Each section says where the two differ.

At a glance

ProblemWhat to doWith Event Gateway
TimeoutsReturn a 2xx within 12 seconds; process in a background jobAccepts the delivery at once, then queues it for your service
DuplicatesRecord the body's id, or a key you add to a trigger payloadDrops repeats before they reach your handler
OrderingCompare time for each subject, or fetch current stateLogs every event and attempt, so you can see arrival order when debugging
RetriesReturn a retryable status, watch for circuit broken, reconcile after an outageAccepts deliveries while your service is down, retries them and lets you replay them
Choosing eventsUse an event subscription for fixed data, and a trigger for conditions on ticketsFilters and routes events by header or body
TestingTest with Admin Center or the API, then with real events on localhostForwards to localhost and replays past events
Other limitsPlan for the 16,000 character trigger payload, the 7-day log and no response dataKeeps its own log of events and attempts

Zendesk webhook timeouts and slow handlers

Return a 2xx status within 12 seconds, then do the work in a background job. Zendesk documents no way to change this limit.

Why it happens: Webhook requests have a 12-second timeout, and Zendesk retries one that times out up to five times. A handler that does the work first can finish that work and still be sent the same request again. Repeated failed requests can also trigger the circuit breaker.

Fix it in your handler:

  1. Verify the signature and return a 200 straight away.
  2. Push the event onto a queue and process it in a worker. See why to process webhooks asynchronously.
  3. Keep calls to other APIs out of the request path, because they use up your 12 seconds.

With Event Gateway: Hookdeck accepts each Zendesk delivery immediately, then queues events and delivers them to your service at a rate it can handle, with its own retries. Your handler can take as long as it needs without failing a Zendesk delivery.

Duplicate Zendesk webhooks

Zendesk can send the same event more than once, even when nothing failed. Keep a record of what you've processed and skip repeats.

Why it happens: Zendesk makes a best effort to deliver once but doesn't guarantee it, and says the same action can invoke a webhook several times. Timeout retries add more.

Your options depend on the webhook:

  • Event webhooks have an id in the body, which Zendesk describes as a unique id for the event.
  • Trigger and automation webhooks have no shared id in the body. It holds only what you put there.
  • x-zendesk-webhook-invocation-id is on every request. Zendesk's API groups an invocation's attempts under one id, so it helps with retries, but a second invocation of the same action gets its own id.
  • Signatures are what Zendesk's docs suggest for spotting duplicates. The signed content includes the timestamp, and the docs don't say whether a duplicate keeps the same timestamp, so don't make a signature your only check.

Fix it in your handler:

  1. For event webhooks, track each id you process, using a database table with a unique constraint on it.
  2. For trigger webhooks, add a key to the payload you write, such as {{ticket.id}} with {{ticket.updated_at_with_time}}, and choose one that suits how often your trigger fires.
  3. Make every handler idempotent: the same result however many times it runs.
async function processZendeskEvent(event) {
  const firstTime = await db.processedEvents.insertIfAbsent(event.id);
  if (!firstTime) return;

  await handleEvent(event);
}

With Event Gateway: Deduplication can drop repeats by a field in the body or a header before they reach your handler, so id works for event webhooks and any key you added works for trigger webhooks.

Zendesk webhook ordering

Zendesk doesn't guarantee delivery order, so don't let arrival order decide which change wins.

Why it happens: When an event, trigger or automation invokes a webhook, Zendesk queues a job and runs it later. Jobs run independently, so there's no guaranteed order, and delivery can be delayed. Retries can also bring an older event in after a newer one.

One action can also produce several ticket events. They share event.meta.sequence.id, and position gives their place in the sequence, though Zendesk doesn't guarantee that order either.

Fix it in your handler:

  1. For event webhooks, store the latest time you've applied for each subject and event type, and ignore anything older.
  2. For trigger webhooks, include an updated time in your payload and compare that instead.
  3. When the order matters, treat the webhook as a signal and fetch the current record from Zendesk's API.

With Event Gateway: Hookdeck doesn't reorder events, but it logs every event and attempt, so you can see what arrived when. Delivering at a rate you set also keeps a burst from reaching your workers all at once.

Zendesk webhook retries, the circuit breaker and lost events

Zendesk retries only some failures, and it describes no way to resend a request. Plan to reconcile after an outage.

Why it happens: Zendesk's retry rules are:

ResultRetry
409Up to three times
429 or 503Only with a retry-after value under 60 seconds
TimeoutUp to five times
Anything else, such as 500, 502 or 504None listed

Repeated failures don't deactivate a webhook, but they can trigger the circuit breaker. It applies when 70% of a webhook's requests fail within five minutes, or when it gets more than 1,000 error responses in five minutes. It doesn't apply if the webhook sends fewer than 100 requests in that time.

While it's on, the webhook can't send for five seconds. The next request then decides what happens: another error starts a new five-second pause, and a success resets the breaker. Invocations during a pause are logged as circuit broken, and Zendesk says an event may not be delivered at all in this case.

Zendesk's docs also describe no failure alerts. The activity log is where failures show up, and it covers the last seven days.

Fix it in your handler:

  1. Return a 2xx once the event is safely queued. If you can't accept it, return a 503 with a retry-after under 60 seconds, because a plain 500 isn't on Zendesk's retry list.
  2. Poll GET /api/v2/webhooks/{webhook_id}/invocations on a schedule and alert on failures. filter[status] accepts values including failed, timeout and circuit broken.
  3. Use GET /api/v2/webhooks/{webhook_id}/invocations/{invocation_id}/attempts to see what happened to one invocation.
  4. After an outage, reconcile from Zendesk's APIs, such as incremental exports or search, because there's nothing to replay.

With Event Gateway: Hookdeck accepts Zendesk's requests while your service is down, so your errors don't count against the circuit breaker. It holds each event, retries on a schedule you set and lets you replay failures individually or in bulk once you've recovered.

Choosing Zendesk events and connection methods

Pick the connection method first, because a webhook can't use both and you can't change it later.

Why it matters: The two methods trade conditions for coverage.

Subscribed to Zendesk eventsConnected to a trigger or automation
Fires onEvery occurrence of the event types you listTicket activity that meets your conditions
CoversTickets, users, organizations, help center, community, agents, messaging and moreTickets
HTTP methodPOST onlyGET, POST, PUT, PATCH or DELETE
FormatJSON onlyJSON, XML or form-encoded
ContentFixed schema with typeA body you write for POST, PUT or PATCH with JSON or XML, otherwise URL parameters
Signature headersYesYes

Do this:

  1. Use an event subscription for system-wide data you want to sync, and subscribe only to the types you process. Zendesk documents no filter on a subscription, so anything else has to be dropped in your handler.
  2. Use a trigger or automation when you need conditions on tickets or a specific payload. Set subscriptions to conditional_ticket_events, then define the content in the notification_webhook action.
  3. Return a 2xx for events you ignore, so they don't count as errors toward the circuit breaker.

Zendesk's developer pages still point ticket activity at triggers, while its event type reference and help article list ticket events for subscriptions too, so tickets can use either method.

With Event Gateway: Hookdeck can filter events by any field in the body or headers and route them to different destinations, which gives event subscriptions the filtering Zendesk doesn't offer.

Testing Zendesk webhooks

Send a test request first, then test with real events. A test request tells you your URL works, but not how your handler copes with real data and failures.

Why it matters: Signing secrets don't exist until a webhook is created, so a test during creation is signed with a static secret. Testing an existing webhook uses that webhook's real secret. A handler that passes while creating the webhook needs the real secret before it passes afterwards.

Do this:

  1. Use Test webhook in Admin Center, which lets you pick an event, edit the request and see the response, or call POST /api/v2/webhooks/test.
  2. Keep verification on while you test, so the code path you test is the one you ship. Verify Zendesk webhooks has the static secret and where to find the real one.
  3. Run through real activity in a sandbox: ticket creation and updates, user and organization changes, and bulk updates.
  4. Check your failure handling too: an endpoint that returns errors, the circuit breaker's circuit broken log entries and duplicate requests.
  5. Note that trial accounts are limited to 10 webhooks and 60 invocations per minute.

With Event Gateway: The Hookdeck CLI gives you a stable public URL to register with Zendesk and forwards events to your machine, and you can replay past events without creating new tickets. Zendesk webhook events has sample payloads to start from.

Other Zendesk webhook limits

Plan for the size limit on trigger payloads, the short activity log and one-way requests.

16,000 characters for a trigger payload. The payload or URL parameters you define for a trigger or automation can't exceed this. Zendesk publishes no limit for event subscription bodies.

Trigger placeholders can render empty. Placeholders insert ticket data when the request is built, and Liquid markup is available for conditions and loops, so handle fields that may be blank. Escape values carefully when you build JSON by hand, and test with real ticket text.

Seven days of history. The activity log covers the last seven days. If you need a longer audit trail, export invocations on a schedule or keep your own log.

No response data in a trigger. Webhook jobs are queued and run separately, and Zendesk describes no way to use your endpoint's response in a trigger or automation. To write data back, your service calls Zendesk's API, or you can use Zendesk Integration Services for a two-way flow. Zendesk warns against using a webhook to update tickets directly, because it can cause race conditions and rate limit errors.

Custom headers. You can add up to five custom headers on an HTTPS endpoint, but don't put credentials in them. Zendesk's API key, basic and bearer authentication options are for that, and they're separate from the signature.

With Event Gateway: Hookdeck keeps a searchable log of events and attempts, and its transformations can reshape a body before it reaches your service.

Keep Zendesk's circuit breaker from dropping your events.

Hookdeck Event Gateway accepts each Zendesk delivery, verifies its signature, queues it and retries to your service on your schedule, and keeps a log of every attempt so you can replay what failed.

Not affiliated with Zendesk. Facts are compiled from public sources and linked inline.