At-Least-Once vs. Exactly-Once Webhook Delivery Guarantees

Webhook delivery guarantees determine how your system handles the fundamental question every distributed system must answer: how many times can an operation be executed? The answer you choose shapes your infrastructure's reliability, complexity, and how you write your application code.

This guide explains the differences between at-least-once and exactly-once semantics, their trade-offs, and how to implement each approach in production webhook infrastructure.

Understanding delivery guarantees

When a webhook provider sends an event to your endpoint, several things can go wrong: your server might be temporarily down, the network might drop the connection, or your application might crash after receiving the payload but before processing it. How your infrastructure handles these failures defines its delivery guarantee.

At-most-once delivery

The simplest approach: send the webhook once and don't retry on failure. If the delivery fails, the event is lost. This works for non-critical notifications where occasional data loss is acceptable. Analytics pings are a good example, but it's rarely appropriate for production webhook systems.

At-least-once delivery

The most common approach in production systems is at-least-once delivery. The provider keeps retrying until it receives a successful acknowledgment (typically a 200 OK response). You're guaranteed to receive every event, but you might receive the same event multiple times.

This happens because acknowledgments themselves can fail. Consider: your server processes a webhook, saves the data, then crashes before returning the 200. The provider never received confirmation, so it retries. Now you've processed the same event twice.

Exactly-once delivery

The reality is exactly-once delivery is impossible in distributed systems.

This isn't a limitation of current technology but a mathematical impossibility proven by the Two Generals Problem. Two parties communicating over an unreliable channel can never be certain they're in perfect agreement because every acknowledgment requires its own acknowledgment, creating infinite regress.

So what can you do instead? Exactly-once processing: at-least-once delivery combined with idempotent consumers that ensure duplicate deliveries don't cause duplicate effects. This is the practical solution used by Stripe, Kafka, AWS, and virtually every reliable webhook provider, including Hookdeck.

At-least-once: how it works

At-least-once delivery requires two components: persistence and acknowledgment.

Persistence before acknowledgment

The provider stores the webhook event in durable storage before returning a 200 OK to the originating service. This prevents the scenario where your server crashes between acknowledgment and processing—the event survives in storage and can be redelivered.

Retry logic

When delivery fails (timeout, network error, or non-2xx response), the provider schedules a retry. Common retry strategies include:

Exponential backoff: Delays between retries increase exponentially (1s, 2s, 4s, 8s...), reducing load on struggling endpoints while still ensuring eventual delivery.

Linear intervals: Fixed delays between retries, useful when you expect problems to resolve quickly.

Custom schedules: Configurable retry windows that match your application's recovery patterns.

Most providers limit total retry attempts and retry windows (typically 1-7 days depending on the provider), after which events move to a dead letter queue for manual recovery.

Implementation example

Here's the basic flow for implementing at-least-once delivery on the receiving side:

app.post("/webhook", async (req, res) => {
  // 1. Immediately acknowledge receipt
  res.status(200).send("OK");

  // 2. Queue for async processing
  await queue.add("process-webhook", {
    payload: req.body,
    receivedAt: Date.now(),
  });
});

// Process asynchronously
queue.process("process-webhook", async (job) => {
  // This may run multiple times for the same event
  await processEvent(job.data.payload);
});

The critical insight: responding with 200 immediately and processing asynchronously. Webhook providers typically timeout requests after 5-60 seconds. If your processing takes longer than that window, the provider retries, leading to duplicate processing.

Exactly-once processing: making duplicates safe

Since you can't prevent duplicate deliveries, the solution is making your processing idempotent (meaning running the same operation multiple times produces the same result as running it once).

Using natural keys

The simplest approach uses unique identifiers from the event itself:

async function processOrder(event) {
  const orderId = event.data.order_id;

  // Attempt insert with unique constraint
  try {
    await db.orders.insert({
      id: orderId,
      ...event.data,
    });
  } catch (error) {
    if (error.code === "UNIQUE_CONSTRAINT_VIOLATION") {
      // Already processed, safe to ignore
      return;
    }
    throw error;
  }
}

This works when your event contains a natural unique identifier (order IDs, transaction IDs, etc.) and your database supports unique constraints.

Using idempotency keys

When natural keys aren't available, generate and store idempotency keys:

async function processEvent(event, idempotencyKey) {
  // Check if already processed
  const existing = await cache.get(`processed:${idempotencyKey}`);
  if (existing) {
    return; // Already handled
  }

  // Process the event
  await performAction(event);

  // Mark as processed with TTL matching retry window
  await cache.set(`processed:${idempotencyKey}`, true, {
    ttl: 72 * 60 * 60, // 72 hours
  });
}

The TTL should match or exceed your provider's retry window. If they retry for 48 hours, your deduplication cache needs to persist at least that long.

Handling concurrent duplicates

Race conditions occur when two identical webhooks arrive nearly simultaneously:

async function processEventSafely(event, idempotencyKey) {
  // Use atomic operations where possible
  const wasSet = await redis.set(
    `processing:${idempotencyKey}`,
    "in-progress",
    "NX", // Only set if not exists
    "EX",
    300 // 5 minute lock
  );

  if (!wasSet) {
    // Another process is handling this event
    return;
  }

  try {
    await performAction(event);
    await redis.set(`processed:${idempotencyKey}`, "done", "EX", 259200);
  } finally {
    await redis.del(`processing:${idempotencyKey}`);
  }
}

Database transactions

For database operations, use transactions with unique constraints:

BEGIN;

-- This fails if payment already processed
INSERT INTO processed_payments (payment_id, webhook_id)
VALUES ($1, $2);

-- Only executes if insert succeeded
UPDATE accounts SET balance = balance + $3
WHERE id = $4;

COMMIT;

The unique constraint on payment_id prevents duplicate processing even under concurrent load.

Trade-offs: choosing your approach

At-least-once without idempotency

Pros:

  • Simple to implement
  • No additional storage for deduplication
  • Works immediately

Cons:

  • Duplicates cause real problems (double charges, duplicate notifications, inflated analytics)
  • Only suitable for inherently idempotent operations

Use when: Operations are naturally safe to repeat, such as updating a record to a final state, overwriting a cache, or logging events where duplicates are acceptable.

At-least-once with idempotency

Pros:

  • Achieves exactly-once semantics at the application level
  • No distributed transaction complexity
  • Standard approach used by major platforms

Cons:

  • Requires additional storage (cache or database)
  • More complex application logic
  • Cache/database becomes a dependency

Use when: Most production webhook systems. Financial operations, user-facing notifications, any case where duplicates would be visible or harmful.

Exactly-once (if your provider offers it)

Some infrastructure providers offer deduplication at the delivery layer, handling duplicates before they reach your application.

Pros:

  • Simpler application code
  • Deduplication handled at infrastructure level
  • Reduced load on your systems

Cons:

  • Still requires idempotent handlers as a fallback
  • Deduplication is typically best-effort, not guaranteed
  • May not catch all edge cases

Use when: Available and your provider's deduplication strategy matches your needs. Always implement idempotency as a safety net.

Important considerations

Webhooks can arrive out of order

Providers don't guarantee delivery order. A payment confirmation might arrive before the order creation webhook, or an update event might precede the create event. Design your handlers to be order-independent, or implement ordering logic that can buffer and resequence events.

Don't rely solely on webhooks

Webhooks should be one part of your data synchronization strategy, not the entire strategy. Implement periodic reconciliation jobs that fetch data directly from the provider's API to catch any events that might have been missed, especially after outages or configuration changes.

When exactly-once matters most

Critical operations where duplicates are unacceptable:

  • Financial transactions: Double-charging a customer is catastrophic
  • Inventory management: Decrementing stock twice oversells products
  • User communications: Nobody wants two identical emails
  • External API calls: Rate limits, costs, and side effects multiply with duplicates

Operations where at-least-once is sufficient:

  • Overwriting state: Setting status = 'complete' twice is harmless
  • Append-only logs: Where duplicates can be filtered later
  • Cache invalidation: Invalidating twice just means an extra cache miss

How Hookdeck supports both approaches

Hookdeck's event gateway operates on an at-least-once delivery guarantee with infrastructure-level features that help you achieve exactly-once processing.

Automatic retries

Configure retry behavior matching your application's recovery patterns. Options include exponential backoff, linear intervals, or custom schedules with retry windows up to a week and 50 delivery attempts.

Deduplication rules

Hookdeck can deduplicate events before they reach your destination using several strategies:

  • Exact match: Drops identical payloads (useful for retry storms)
  • Include fields: Deduplicate based on specific fields like request IDs
  • Exclude fields: Ignore events where only non-essential fields changed

For example, configuring X-Shopify-Event-Id as the deduplication key ensures each unique Shopify webhook processes only once, regardless of how many times Shopify attempts delivery.

Note that deduplication windows are configurable between 1 second and 1 hour. This makes deduplication effective for handling retry storms and near-simultaneous duplicates, but it's not a replacement for application-level idempotency. You should always implement idempotent handlers as your primary defense.

Idempotency headers

Every event includes headers like X-Hookdeck-EventID that you can use as idempotency keys in your application. Even if Hookdeck delivers the same event twice due to network conditions, your code can use this ID to detect and skip duplicates.

Dead letter queue management

Failed events aren't lost, they're instead tracked as issues with full payload visibility. You can retry events individually, in bulk, or set up automatic retry policies for specific failure types.

Event persistence

Events are persisted to durable storage before returning a 200 to providers. If your server crashes between acknowledgment and processing, the event is safe and can be redelivered.

Conclusion

The practical reality of webhook infrastructure is this: exactly-once delivery is impossible, but exactly-once processing is achievable. Build your systems on at-least-once delivery with idempotent handlers, and you'll have reliable webhook processing without the complexity of distributed transactions.

Choose your implementation based on the consequences of duplicates. For critical operations, invest in robust idempotency. For naturally safe operations, the simpler path works fine. And use infrastructure-level deduplication as an optimization, not a replacement for application-level safety.

Resilience Thin Events