Webhooks Fetch Before Process Pattern

When building systems that consume webhooks, a common and robust pattern is to treat the incoming webhook as a notification, and then fetch the full, up-to-date resource from the provider's API before any processing takes place. This is known as the "Fetch Before Process" pattern.

This approach provides a strong defense against common issues in event-driven architectures, such as receiving events out of order or handling duplicate events. By fetching the latest state of a resource, you ensure your system is always working with the most current data, which simplifies your logic and enhances reliability.

What is the 'Fetch Before Process' pattern?

The core concept of the pattern is to treat the incoming webhook event as a signal to trigger a process, but instead of using the data in the webhook payload, you first make an API call to fetch the full, up-to-date resource. This ensures you are always working with the most current data.

Why 'Fetch Before Process' is a reliable pattern

Webhook-driven systems are inherently asynchronous, which means you must design for challenges like:

• Event Ordering: You can't guarantee the order in which you receive webhooks. For example, an updated event might arrive before the created event for the same resource, leading to processing errors.
• Duplicate Events: Most webhook providers have an "at-least-once" delivery guarantee. This means you will occasionally receive the same event more than once, and your system must be ableto handle this without causing unintended side effects. This is the principle of idempotency.

Fetching the resource from the API just before you process it helps solve these problems. It allows you to:

1. Work with the most current data: You're no longer relying on the data snapshot in the webhook payload, which might be stale.
2. Validate the resource's state: You can confirm the resource is in the expected state before taking action.
3. Simplify idempotency: Your processing logic is based on the actual state of the resource, not the event itself.

Some providers, like Stripe, are beginning to formalize this pattern by offering "thin" events. These are webhook payloads that contain only the resource ID, forcing the consumer to fetch the details from the API.

The risk: API rate limiting at scale

While the "Fetch Before Process" pattern is reliable, it has a significant challenge as your application scales: API rate limits.

Imagine a sudden burst of activity—like a flash sale or a large data import—that generates thousands of webhooks in a short period. If your system attempts to fetch the resource for every single incoming event, you can easily generate a high volume of API requests per second. This can exceed the provider's rate limits (e.g., Stripe's is typically 100 read requests per second), causing them to respond with 429 Too Many Requests errors.

When you hit a rate limit, your system can fail to process events, leading to data inconsistencies and a poor user experience.

A resilient approach: Decouple ingestion from processing

To get the benefits of the "Fetch Before Process" pattern without the risk of hitting rate limits, you can decouple webhook ingestion from processing using a queue.

Here's how this more resilient architecture works:

1. Receive and Queue: An event gateway (like Hookdeck) receives all incoming webhooks and places them into a queue.
2. Throttle Delivery: You configure the gateway to deliver events to your application at a controlled rate that stays safely within your provider's API limits.
3. Fetch and Process: Your application code receives the throttled events and can now safely fetch the resource from the API before processing it.

By introducing a queue, you absorb sudden spikes in webhook volume and ensure your application processes events at a steady, manageable pace. This gives you the reliability of the "Fetch Before Process" pattern while protecting your system from being overwhelmed.