Webhook Ordering: Why It's Hard and How to Handle It

When you're building webhook integrations, ordering seems like it should be straightforward. Events happen in a sequence, so webhooks should arrive in that same sequence. In practice, guaranteeing webhook order is surprisingly difficult, and attempting to enforce it often creates more problems than it solves.

This guide explains why webhook ordering is challenging, when it actually matters, and practical strategies for building systems that handle out-of-order delivery gracefully.

Why ordering seems important

Consider an e-commerce platform where orders reference inventory items. When a merchant creates a new product and immediately sells a unit, two webhooks fire in quick succession: product.created followed by order.placed.

The problem emerges on the receiving end. If the order webhook arrives first, the consuming system tries to decrement inventory for a product that doesn't exist in its database yet. The webhook handler fails with a foreign key violation, or worse, silently drops the order data.

This scenario illustrates why developers instinctively reach for ordering guarantees. The logic seems simple: if events occur sequentially, deliver them sequentially.

Why ordering guarantees are difficult

Guaranteeing webhook order faces three fundamental challenges: throughput limitations, handling failures, and controlling what happens after delivery.

The throughput bottleneck

Ordered delivery requires serialization. The provider must send one webhook, wait for acknowledgment, then send the next. This creates a hard throughput ceiling tied directly to consumer response time.

If your endpoint averages 150ms response time, the maximum throughput is roughly 6-7 webhooks per second per endpoint. For a platform sending webhooks to thousands of customers, this might be manageable. For a single high-volume integration receiving thousands of events per minute, it's a dealbreaker.

The math gets worse with retries. If the provider waits 30 seconds before retrying a failed delivery, every failure creates a 30-second pause in the entire stream. During a partial outage where 10% of requests fail, throughput collapses.

Parallel delivery solves the throughput problem but abandons ordering. Most providers choose parallelism because the alternative of telling customers they can only receive a handful of webhooks per second is commercially unviable.

The failure dilemma

When a webhook delivery fails, you have two options, and neither is satisfying.

Option one: Block the queue. Hold all subsequent webhooks until the failed one succeeds. This guarantees order but creates a fragile system. A single malformed payload or a bug in one webhook handler blocks every other webhook. A warehouse stops receiving shipment notifications because a low-priority preferences.updated webhook keeps timing out.

Option two: Skip and continue. Deliver subsequent webhooks while retrying the failed one. This maintains throughput but abandons ordering. The order.placed webhook delivers successfully while product.created is still retrying, exactly the scenario you were trying to prevent.

Most webhook providers choose option two because blocking is catastrophic at scale. But this means ordering is only guaranteed when everything works perfectly, which is precisely when you don't need guarantees.

If consumers need to handle out-of-order scenarios when errors occur, they may as well handle out-of-order scenarios in general because you'd end up with the same code either way. The ordering guarantee provides no practical value.

The processing problem

Even if a provider accepts the throughput penalty and delivers webhooks sequentially, they may not be processed in order. This issue is entirely outside the sender's control.

Imagine two webhook handlers with different performance characteristics:

def handle_product_created(payload):
    # Sync with external inventory system (slow network call)
    inventory_service.sync_product(payload)  # Takes 800ms

    # Store locally
    db.products.insert(payload)
    return 200

def handle_order_placed(payload):
    # Simple database write (fast)
    db.orders.insert(payload)
    return 200

The first webhook triggers an external API call that takes 800ms. The second webhook is a simple database insert completing in 15ms. Even when product.created arrives first, the order persists to the database before the product exists.

You might think: wait for each webhook to finish processing before sending the next one. But this compounds the throughput problem discussed above. If handlers average 200ms, you're capped at 5 webhooks per second per endpoint. For high-volume integrations, this is unworkable.

There's a deeper problem too. Best practice for webhook consumption is to validate, enqueue, and return immediately.

This pattern is essential for reliability. It prevents timeouts and allows the consumer to control processing rate. But it means the webhook provider has no visibility into actual processing order. Returning 200 doesn't mean the event was processed; it means it was acknowledged. Unless the consumer's queue processes items strictly in order (and waits for each to complete), the same race conditions apply.

When ordering actually matters

Before implementing complex ordering solutions, determine whether you genuinely need them. Many ordering requirements dissolve under closer examination.

State convergence matters more than event order. If your system eventually reaches the correct state regardless of event order, ordering is a convenience rather than a requirement. Most create/update/delete operations fall into this category.

Ordering within an entity matters more than global ordering. You rarely need all webhooks ordered globally. More commonly, you need events for the same entity ordered: all updates to product SKU-1234 should apply in sequence, but SKU-1234 and SKU-5678 can process independently.

Final state often matters more than intermediate states. If a product's price changes three times in rapid succession, do you need to process all three updates in order? Or do you just need the final price?

Strategies for handling out-of-order delivery

Rather than fighting webhook infrastructure to guarantee ordering, design your systems to handle out-of-order delivery gracefully.

Include freshness indicators

Every webhook payload should include information that lets consumers determine whether an event is newer or older than what they already have:

{
  "event_type": "product.updated",
  "data": {
    "id": "prod_abc123",
    "name": "Wireless Headphones",
    "price": 79.99,
    "modified_at": "2024-01-15T10:30:00Z",
    "revision": 17
  }
}

Consumers compare the incoming revision against their stored value:

def handle_product_updated(payload):
    product_id = payload['data']['id']
    incoming_revision = payload['data']['revision']

    existing = db.products.find(product_id)

    if existing and existing.revision >= incoming_revision:
        # Already have newer data, acknowledge but skip processing
        log.info(f"Skipping outdated event for {product_id}")
        return 200

    db.products.upsert(payload['data'])
    return 200

This pattern handles out-of-order delivery, duplicate delivery, and replay scenarios elegantly.

Use timestamps as a fallback

When revision numbers aren't available, timestamps serve a similar purpose:

def handle_event(payload):
    entity_id = payload['data']['id']
    event_time = parse_datetime(payload['data']['modified_at'])

    existing = db.find(entity_id)

    if existing and existing.modified_at >= event_time:
        return 200  # Skip outdated event

    db.upsert(payload['data'])
    return 200

Be cautious with timestamps across distributed systems. Clock drift between servers can cause incorrect ordering decisions. Revision counters or sequence numbers are more reliable when available.

Design for idempotency

Idempotent handlers produce the same result whether an event is processed once, twice, or ten times. This property is essential for reliable webhook consumption regardless of ordering concerns.

Use event identifiers to track what you've already processed:

def handle_webhook(payload):
    event_id = payload['event_id']

    if db.processed_events.exists(event_id):
        return 200  # Already handled

    with db.transaction():
        process_event(payload)
        db.processed_events.insert(event_id)

    return 200

Handle missing dependencies gracefully

When webhooks reference entities that don't exist yet, you have several options:

• Retry with backoff. Return a failure status and let the webhook provider retry. The dependent entity may exist by the next attempt.
• Create placeholder records. Insert a minimal stub that gets enriched when the full entity arrives.
• Queue for deferred processing. Store events that can't be processed immediately and retry them periodically.

Fetch current state instead of trusting payloads

Instead of relying on webhook payloads containing complete and current data, send minimal notifications that prompt consumers to fetch the latest state:

{
  "event_type": "product.updated",
  "data": {
    "id": "prod_abc123",
    "updated_fields": ["price", "inventory_count"]
  }
}

The consumer fetches current state via API:

def handle_product_updated(payload):
    product_id = payload['data']['id']

    # Fetch authoritative current state
    current_data = api_client.get_product(product_id)

    # Always working with latest data regardless of webhook timing
    db.products.upsert(current_data)
    return 200

This approach guarantees you're always working with current data, regardless of webhook delivery order. The trade-off is increased API load and added latency in processing.

Alternative architectures

When ordering requirements are genuinely strict, consider architectures purpose-built for sequential delivery.

Event streaming

Message streaming platforms like Apache Kafka provide ordering guarantees within partitions. Events for the same entity route to the same partition and process in sequence.

Consumers process each partition sequentially, guaranteeing order for events sharing a key. This is a fundamental architectural shift from webhooks but provides strong ordering guarantees where needed.

Polling with cursors

Instead of push-based webhooks, consumers pull from an events API.

The provider maintains event sequence. Consumers control their processing pace and can guarantee sequential processing. Some webhook providers offer this as a companion feature to push-based webhooks.

Batched sequential delivery

Some providers offer endpoints that batch multiple events into single deliveries:

{
  "batch": [
    {"sequence": 1, "type": "product.created", "data": {...}},
    {"sequence": 2, "type": "inventory.adjusted", "data": {...}},
    {"sequence": 3, "type": "order.placed", "data": {...}}
  ]
}

The consumer processes the batch sequentially within a single request. This maintains ordering while reducing HTTP overhead. The trade-off is increased consumer complexity and potentially larger payloads.

Choosing the right approach

The best approach depends on your specific requirements:

For most webhook integrations, designing handlers that tolerate out-of-order delivery is simpler and more robust than attempting to guarantee ordering at the infrastructure level.

Conclusion

Webhook ordering is difficult for three reasons. First, serialized delivery caps throughput at a few requests per second (unworkable for high-volume integrations). Second, failures force providers to choose between blocking (which is fragile) and continuing (which breaks ordering). Third, consumers control what happens after delivery, and processing order depends on handler implementation, not arrival sequence.

Rather than fighting these constraints, build systems that handle out-of-order delivery gracefully. Include revision numbers or timestamps so consumers can detect stale events. Implement idempotent handlers that produce correct results regardless of processing order. Handle missing dependencies through retries, placeholder records, or deferred processing queues.

When strict ordering is genuinely required, consider architectural alternatives: event streaming platforms, cursor-based polling, or batched sequential delivery. These approaches are designed for ordered consumption and avoid the inherent limitations of traditional webhook infrastructure.

The goal isn't perfect ordering. It's building systems that converge to the correct state regardless of the order events arrive.