What Are the Implementation Considerations for Webhook Producers

Webhooks are used to integrate networked applications for the main purpose of transferring information. When application A needs to communicate with application B, A generates a webhook and sends it to B through an HTTP request. This is a one-directional and often fire-and-forget style of communication that terminates once the webhook is received by the destination application. The application that sends the webhook is referred to as the webhook producer, while the application receiving the webhook is called the webhook consumer.

In production environments, this communication is not as simple as I just described. Both webhook producers and consumers require a certain amount of performance and fault-tolerant characteristics in order to function optimally and be reliable.

In this article, we focus on webhook producers. We take a look at important considerations to account for when working with webhook producers, how to improve response time, and what we can do to scale their operations.

^What are webhook producers?

Message producers generate and trigger webhook requests in reaction to an event taking place in the source application’s system. For example, a webhook can be configured to be triggered when a purchase is made in an e-commerce application like Shopify (or a custom e-commerce application).

When the event occurs, a webhook request is generated and sent to a destination application. During the webhook configuration, a URL endpoint on the destination application will be set as the target for the webhook request. This URL is known as the webhook URL.

After the webhook is sent, the producer waits for a response from the consumer. Depending on the response, the producer can confirm that the webhook has been consumed, retry the webhook or simply forget it.

Because webhook producers set a hard limit on the response time for webhooks, it is a recommended best practice to process webhooks asynchronously. One of the common ways to implement asynchronous processing for webhooks is by using a message queueing middleware (often called a message broker).

Most message brokers do not speak the HTTP protocol so to use a message broker with a webhook producer, an API gateway is required to convert the webhook messages from HTTP to the protocol compatible with the message broker being used.

In the next section, we’ll take a look at how we can make this asynchronous setup reliable for our webhooks.

^What are the implementation considerations when dealing with webhook producers?

In this section, we will take a look at the information and considerations we need to take into account when working with webhook producers. Most of the time, as it is with third-party webhook producers like Stripe and Shopify, we will not have full control of how webhooks are generated and triggered.

Understanding how the webhook is triggered, as well as the nature, form, and structure of the webhook request is key to handling the webhook.

Let’s take a look at some of these important factors.

• Request protocol: Webhooks are sent using the HTTP protocol, however you need to know if the webhook is strictly using HTTPS or a mixture of both (although this is rare). Because HTTPS endpoints accept both HTTP and HTTPS requests, the API gateway needs to provide an HTTPS endpoint for your webhook producers to target.
• Request method: Your API gateway’s endpoint needs to support the request method through which your webhooks are sent. From my experience working with webhooks, almost all (third-party) webhook providers that I have worked with make use of the POST method to send webhooks. You can provide an endpoint that supports both the POST and GET methods if you’re going to be receiving a mixture of both request types.
• Automatic and/or manual retries: Certain providers have retry systems (manual, automatic, or both) built into them. A retry system helps resend a webhook when it fails on the side of the consumer. Understanding how reliable a provider’s retry system is (if at all) and how it operates helps in knowing if you should build your own retry system into your webhook infrastructure and how elaborate you want it to be.
• Request timeout: As stated earlier, webhook producers set a hard limit on the request timeout for each webhook. The response time for every webhook that hits your gateway should always be below this value no matter the number of concurrent webhooks your gateway needs to process. For example, if the timeout for each webhook is 5 seconds, the gateway response time should not be 4 seconds for each webhook. It is important to stay below the webhook producer’s request timeout threshold even as the number of concurrent webhooks grows.
• Message format and structure: The format (JSON or XML) and message structure of the webhook both need to be factored in to how it is handled within the API gateway. You also need to understand the webhook payload and headers structure for necessary transformations like adding a Trace ID to the request for the purpose of monitoring the webhook.
• Verification: Some webhook providers require that the client goes through a verification process before it can access the webhook information. A common verification process involves symmetric encryption using a secret key. If verification is required, the API gateway should be provided with all the information necessary to pass it. You can also choose to defer the verification responsibility to the actual consumers.

^What are the scalability considerations when handling a webhook producer’s load?

One of the requirements of a reliable webhook infrastructure is the ability to keep adding more of the same type of webhook producer instances and scale with an increasing number of producers. For example, imagine you have a payment service that processes payments for your Shopify store. Your Shopify store sends a webhook to the payment service when a purchase is made. Let’s assume your business grows and you need to spin up more Shopify stores for customers from other continents or countries. Instead of just one, you now have multiple Shopify store instances sending webhooks to the same payment service.

The first step you need to take is to ensure that producers are completely decoupled from consumers. This means that producers are not meant to know where the consumers are or how they operate. Our API gateway already takes care of this by being the only destination for producers to target their webhook requests. Details of how the API gateway handles the webhooks are also abstracted from the producer. All a producer does is send a webhook request and receive a successful response within its timeout limit. From that point, its work is done.

The second step is to place a load balancer in between your webhook producers and a pool of API gateway servers. This allows you to spread the volume of webhooks received from your producers amongst these gateway instances. With this setup, you improve the response time for your webhooks requests and the overall reliability of your infrastructure.

As webhook volume continues to increase, another component in the asynchronous setup that can get overwhelmed is the message broker. Luckily, each broker technology gives you the ability to deploy multiple broker instances in a broker cluster. This helps boost the ingestion of webhooks for brokers as there are more queues to write to. The queue lengths of brokers are not easily exceeded, there are more network sockets to work with, and the queue’s throughput is improved.

^Conclusion

Webhook producers are the source of the webhooks your infrastructure will have to deal with, and therefore scaling your infrastructure to handle webhooks begins with producers. In this article, you have seen how scaling webhook producers starts with decoupling the producers from the consumers and ingesting them into a message queue, and then through an API gateway for asynchronous processing. You have also learned all the important factors that need to be taken into account when designing your asynchronous processing strategy.

These factors and implementation best practices are not exhaustive in themselves, but they do give you a great start to scaling your webhook producers.

Happy coding!