Retries

A retry is a new delivery attempt for an event. Retries can be triggered manually on any event, scheduled or automatically on failed events.

Events are limited to 50 automatic retries, but can be manually retried as many times as you like.

Manually retry events

Manually retrying events allows you to re-attempt delivery to a destination.

Manual retries are useful when troubleshooting, testing, or after resolving an issue on the destination. If a manual retry is successful, any further automatic retries on that event will be canceled – otherwise, future retries will continue as planned.

Manual retries can be triggered individually or in bulk.

Retry a single event

{
  "id": "evt_njCeMZHzNhJWdkkZ7PLCfEw5",
  "team_id": "tm_nlcetVe8k1lMAY0KR0OxNuHr",
  "webhook_id": "web_UfM5ZY2wxLD6sLhDaKMcx5wp",
  "source_id": "src_9YmY0SHklrv6zBQz78cKGwSW",
  "destination_id": "des_fYYCay655IHwbOMhIaUXtNq2",
  "attempts": 4,
  "response_status": 422,
  "last_attempt_at": "2021-08-02T13:52:16.439Z",
  "next_attempt_at": null,
  "successful_at": "2021-08-01T21:56:35.181Z",
  "updated_at": "2021-08-02T14:12:56.321Z",
  "created_at": "2021-08-01T21:44:57.513Z",
  "status": "QUEUED",
  "event_data_id": "evtreq_HlJhb39nsjIP0FDsU3SY0fo7",
  "cli_id": null,
  "data": {
    "headers": {
      "content-length": "20",
      "x-original-url": "/e/src_9YmY0SHklrv6zBQz78cKGwSW",
      "cookie": "",
      "x-client-version": "FirebaseCLI/undefined",
      "content-type": "application/json",
      "user-agent": "FirebaseCLI/undefined"
    },
    "body": {
      "test": true
    },
    "query": "",
    "parsed_query": {},
    "path": "/"
  }
}

Hookdeck will now attempt to deliver the event to its destination. If it succeeds, any scheduled automatic retries for that event will be canceled.

Retry many events

Bulk retrying events may impact your delivery latency, as retries are queued alongside new events.

{
  "query": {
    "source_id": [
      "src_oDhfDrVAZV6OIPdBzcD97of2"
    ]
  }
}

{
  "id": "blkr_6dxoWIomd8DQdH",
  "team_id": "tm_UkFg3CHYN50M",
  "query": {
    "rejection_cause": [
      "SOURCE_DISABLED"
    ]
  },
  "updated_at": "2022-05-13T09:06:03.983Z",
  "created_at": "2022-05-13T09:05:55.394Z",
  "completed_at": null,
  "cancelled_at": "2022-05-13T09:06:03.982Z",
  "estimated_batch": 4,
  "processed_batch": 2,
  "estimated_count": 28,
  "completed_count": 8,
  "failed_count": 0,
  "number": 15,
  "in_progress": false,
  "progress": 0.29
}

Hookdeck will now attempt to deliver the events. For each event whose delivery succeeds, any scheduled automatic retries will be canceled.

Automatically retry events

With automatic retries, Hookdeck will continue to attempt to delivery events that have previously failed to be delivered.

Each automatic retry is counted as a new delivery attempt. Once an attempt succeeds, further automatic retries on that event are canceled.

Events are limited to 50 automatic retries, but can still be manually retried after that limit is hit.

When automatic retries occur

Automatic retries occur in the following scenarios:

• Timeout: If a destination does not respond within 60 seconds.
• Network error: If a destination is unreachable due to network issues.
• HTTP status code: If a destination responds with an HTTP status code that has been configured to retry. By default, any non-2xx status code will trigger a retry, but you can configure this behavior to only retry specific status codes.

Configuring automatic retries

Configure a connection's automatic retries by setting its retry rule. There you can define the number of automatic retries Hookdeck will attempt after an event fails, the time interval between each attempt, the retry strategy and the response HTTP status codes to apply the retry rule to.

The retry strategy can be either linear or exponential. With a linear strategy, retries occur at regular intervals; with an exponential strategy, each retry is delayed twice as long as the previous (1 hour, 2 hours, 4 hours, etc.).

The response status codes can be used to specify which status codes should trigger a retry. For example, if you set the retry rule to apply to status codes 500-599, Hookdeck will retry events that receive any of those status codes. The status codes supports inclusion, exclusion, ranges and greater than/less than comparisons (>=, <=, >, <).

For example, to retry all 5xx errors, you can use 500-599 or >=500 and to retry all 5xx errors except 501 you can use 500-599, !501.

In the case of a conflicting HTTP status code, Hookdeck will take the last matching statement. For example, 500, !500 will not retry 500s.

Retry strategy examples

• Linear: If the retry rule is set to retry 3 times with a 10 minute interval, Hookdeck will retry the event at 10, 20, and 30 minutes after the initial failure.
• Exponential: If the retry rule is set to retry 3 times with a 10 minute interval, Hookdeck will retry the event at 10, 20, and 40 minutes after the initial failure.

Identifying retries

When a HTTP request is made to your destination, Hookdeck will include a x-hookdeck-will-retry-after header in the request headers. This header indicates the number of seconds that the next retry will be scheduled in once your server responds, assuming that a retry is applicable. If there's no x-hookdeck-will-retry-after then the request is the last automatic retry.

Scheduling a automatic retry

Your destination can respond with a Retry-After header to indicate how long Hookdeck should wait before retrying the event. This can be useful to implement custom retry logic or scheduling of retries based on specific errors. The Retry-After header must be an integer value representing the number of seconds to wait before retrying the request, a valid date string as per the HTTP spec or a ISO string.

Returning a value of -1 will cancel any further automatic retries.

If your destination returns a Retry-After header, the value will take precedence over any retry rule. You don't need to set a retry rule in order to use the Retry-After header.

Using the Retry-After header, a retry can be scheduled up to 7 days in the future and at most 50 times.

Viewing the next attempt

To determine when a specific event will be retried, locate the event and note its Next Attempt At property.

Cancel scheduled retries

Canceling an automatic retry will prevent all future scheduled retries from occurring on an event.

{
  "id": "evt_njCeMZHzNhJWdkkZ7PLCfEw5",
  "team_id": "tm_nlcetVe8k1lMAY0KR0OxNuHr",
  "webhook_id": "web_UfM5ZY2wxLD6sLhDaKMcx5wp",
  "source_id": "src_9YmY0SHklrv6zBQz78cKGwSW",
  "destination_id": "des_fYYCay655IHwbOMhIaUXtNq2",
  "attempts": 5,
  "response_status": 422,
  "last_attempt_at": "2021-08-02T13:52:16.439Z",
  "next_attempt_at": null,
  "successful_at": "2021-08-01T21:56:35.181Z",
  "updated_at": "2021-08-02T14:12:56.466Z",
  "created_at": "2021-08-01T21:44:57.513Z",
  "status": "QUEUED",
  "event_data_id": "evtreq_HlJhb39nsjIP0FDsU3SY0fo7",
  "cli_id": null,
  "data": {
    "headers": {
      "content-length": "20",
      "x-original-url": "/e/src_9YmY0SHklrv6zBQz78cKGwSW",
      "cookie": "",
      "x-client-version": "FirebaseCLI/undefined",
      "content-type": "application/json",
      "user-agent": "FirebaseCLI/undefined"
    },
    "body": {
      "test": true
    },
    "query": "",
    "parsed_query": {},
    "path": "/"
  }
}

Best practices when handling retries

Retries are a powerful feature, but they can also lead to unexpected behavior if not handled correctly. Here are some best practices to follow when handling retries:

Make endpoints idempotent

Your webhook handlers should be designed to safely process the same event multiple times without unintended side effects. This is crucial because webhook providers often guarantee "at least once" delivery, meaning duplicates are possible. Hookdeck has the same "at least once" guarantee.

For a detailed guide on achieving idempotency, see How to Implement Webhook Idempotency.

Respond within 60 seconds

Hookdeck times out requests that take longer than 60 seconds to respond, so ensure your webhook handler's logic can complete within that timeframe. Delivery attempts that take longer than 60 will be classified as failed and will be retried according to your retry rules.

In some cases, we can extend the timeout. If you need a longer timeout, please contact support to discuss your requirements.

Monitor and alert failed deliveries

To effectively manage webhook delivery, set up monitoring and alerts for failed deliveries. Using Issues & Notifications, you can configure alerts that trigger on either the first failed attempt (to catch problems early) or the last failed retry attempt (to identify persistent issues after all retries are exhausted). This approach helps you distinguish between temporary hiccups and systemic problems requiring immediate attention.