Taking Advantage of Increasingly Popular Webhooks

Historically, if you as a developer wanted to get real-time information about an occurrence that exposed an API in your system, you would have to poll the API. You would then have to watch and wait for a status change or a 2XX response. It wasn't until 2007 that the term "webhook" was coined by Jeff Lindsay, and a new alternative to polling was born. Instead of the arduous task of manually collecting information, we can now ask the server that controls the resources we’re interested in to notify us whenever a change occurs. Essentially, webhooks are the computer version of the famous Hollywood adage “Don’t call us, we’ll call you!"

This article covers the benefits of webhooks, integration webhook publishers (API platforms), some of the problems related to their utilization, and how to solve them.

Benefits of using webhooks

Eliminates polling

Webhooks were not well known when the term first came on the scene, but have become more popular because of the many benefits they bring. Primarily, webhooks eliminate the need for polling, a task that is painful for both the provider and the consumer. For the provider, polling creates overhead (the client has to request the resource many times before getting the response that they expect). For the consumer, polling requires complex code with retry logic and rate-limiting that engineering teams must maintain.

Simple HTTP request

Webhooks are extremely simple to use and usually come in the form of a straightforward HTTP request initiated by the publisher. There is no dependency on any language or other technology. The HTTP method you will receive the webhook with is generally a POST, and the payload is generally JSON or XML, though it could be any text-based format. Part of what makes webhooks so attractive is that if you’re developing an API, all you have to do is add one endpoint to process incoming webhook requests, as you already have all the infrastructure.

Example of consuming a webhook

Let’s say that you want to receive events related to Shopify on

https://www.myawesomesystem.com/shopify.

After configuring your URL in the publisher’s platform (in this case Shopify), if you’re using node.js and express, you can write:

‍

const express = require("express");
const app = express();
app.use(express.json());
app.post("/test", (req, res) => {
  console.log(req.body);
  console.log(req.body.test);
  res.sendStatus(200);
});
app.listen(8000, () => {
  console.log("listening on 8000");
});

Notice here that we’re using res.sendStatus(200), which is equivalent to res.status(200).send(“OK”). Express 4.x introduced that syntax.

We’ve omitted some details to keep this article simple. However, some of them are worth noting:

1. Shopify verifies the consumer’s SSL certificate and won’t send the payload if the verification fails.
2. Shopify will also send a few request headers along with the HTTP request. They contain the Shopify topic, a sha256 signature so that you can verify the payload, and a couple of other helpful tidbits of information.

API providers who publish webhooks

Now that we have a good idea of how a webhook is consumed, let's go over who publishes them. Platforms that need to offer information about events in real-time will often offer webhooks. Generally, any system that needs to expose state information can do so with a webhook.

Ecommerce

Shopify, for instance, offers webhooks that allow us to know all sorts of events that happen on their platform. Two commonly implemented use cases are inventory synchronization and order fulfillment – Shopify sends a webhook to subscribers with a payload containing information about the purchase. There are many other use cases, for example we have an article on synchronizing your Shopify orders with webhooks.

CI/CD

Github also offers webhooks. Perhaps the most common use case is to get notified when a push occurs on a branch. Those events allow subscribers to start a CI pipeline that builds our application and deploys it.

Communication

Twillio is known for being developer-friendly. Their webhook offering helps create integration based on SMS triggers. There are many other use cases such as automating Vercel Deploys via SMS using Twilio webhooks or building a customer rating app with Twilio webhooks.

Slack also offers the ability to post messages to channels using webhooks that you can create from the platform's UI or its API. This allows you to post really any content in any channel in Slack. All the services we mentioned above that publish webhooks could be integrated with Slack - for instance, we could get notified on Slack of a ticket opened in Jira via webhooks.

Identity management

We can talk about security with Okta offers event hooks. Okta notifies subscribers about anything that happens on their platform – new user subscription, user activation and deactivation, password change, application addition to a user’s profile, and much more. This guide covers Okta’s event hooks and how they work.

Other use cases

Collaboration (Atlassian), customer service (Intercom), email delivery (Mailchimp), payment processing (Stripe)... the list goes on. You can expect to see webhooks for platforms that need to offer information about events as they happen, either in subscribing (incoming) or publishing (outgoing) form.

Problems with webhooks and how to deal with them

The popularity of webhooks is primarily attributable to the fact that they offer an easy, simple solution to a hairy problem. However, like everything, difficulties can always arise. As with anything technical, the amount of effort you put into securing and scaling your webhook's infrastructure is proportional to the impact on your business if you should fail to handle them.

Handling webhook spikes

To handle spikes of incoming webhooks, for example on a holiday where traffic patterns differ from usual, you can implement a queuing system that defers the processing of webhooks payload until a time that you have workers available

Error handling of webhooks

If the publisher does not support retry for requests that fail, or if you miss all of the retries due to an error, the only way to fix this problem is to create a separate ingest point that never fails, log all the data that comes through it, and then forward and retry those requests that fail.

Debugging webhooks

To debug webhook calls that fail on your end, you can log the payload data and assign an ID to each request so that you can later correlate those requests with your internal data, or if that’s not enough, you can develop tooling to analyze webhook payload.

Fanning out webhooks

To get your request in two different places (like a log and an actual action, for instance), you can implement a fan-out pattern to get the request and redirect it to two or three other endpoints. Of course, you’d have to keep track of which endpoint succeeded and which one failed and apply a retry logic.

There are solutions to all these problems, though they take some work to get right.

Conclusion

This article and others in this series were hopefully helpful in understanding how webhooks work and how to build a resilient infrastructure to handle them. Webhooks are, in most cases, an integral infrastructural piece that might lead to losses in case of service interruption. It is therefore essential to build resilience into your projects.

Hookdeck is a feature-complete webhooks handling platform that handles resilience and monitoring for you. It enables you to decide how to route webhooks requests, debug and visualize your payloads, offer insight into errors and trends, retry on delivery failure, hook into your development process with a local proxy, and much more. Try Hookdeck now for free by creating an account here.