Taking Advantage of Increasingly Popular Webhooks

Eliminates polling

Webhooks were not well known when the term first came on the scene, but have become more popular because of the many benefits they bring. Primarily, webhooks eliminate the need for polling, a task that is painful for both the provider and the consumer. For the provider, polling creates overhead (the client has to request the resource many times before getting the response that they expect). For the consumer, polling requires complex code with retry logic and rate-limiting that engineering teams must maintain.

Simple HTTP request

Webhooks are extremely simple to use and usually come in the form of a straightforward HTTP request initiated by the publisher. There is no dependency on any language or other technology. The HTTP method you will receive the webhook with is generally a POST, and the payload is generally JSON or XML, though it could be any text-based format. Part of what makes webhooks so attractive is that if you’re developing an API, all you have to do is add one endpoint to process incoming webhook requests, as you already have all the infrastructure.

Example of consuming a webhook

Let’s say that you want to receive events related to Shopify on

https://www.myawesomesystem.com/shopify.

After configuring your URL in the publisher’s platform (in this case Shopify), if you’re using node.js and express, you can write:

‍

const express = require('express');
const app = express();
app.use(express.json());
app.post('/test', (req, res) => {
   console.log(req.body);
   console.log(req.body.test);
   res.sendStatus(200);
});
app.listen(8000, () => {
   console.log('listening on 8000');
})

Notice here that we’re using res.sendStatus(200), which is equivalent to res.status(200).send(“OK”). Express 4.x introduced that syntax.

We’ve omitted some details to keep this article simple. However, some of them are worth noting:

1. Shopify verifies the consumer’s SSL certificate and won’t send the payload if the verification fails.
2. Shopify will also send a few request headers along with the HTTP request. They contain the Shopify topic, a sha256 signature so that you can verify the payload, and a couple of other helpful tidbits of information.

Ecommerce

Shopify, for instance, offers webhooks that allow us to know all sorts of events that happen on their platform. Two commonly implemented use cases are inventory synchronization and order fulfillment – Shopify sends a webhook to subscribers with a payload containing information about the purchase. There are many other use cases, for example we have an article on synchronizing your Shopify orders with webhooks.

CI/CD

Github also offers webhooks. Perhaps the most common use case is to get notified when a push occurs on a branch. Those events allow subscribers to start a CI pipeline that builds our application and deploys it.

Communication

Twillio is known for being developer-friendly. Their webhook offering helps create integration based on SMS triggers. There are many other use cases such as automating Vercel Deploys via SMS using Twilio webhooks or building a customer rating app with Twilio webhooks.

Slack also offers the ability to post messages to channels using webhooks that you can create from the platform's UI or its API. This allows you to post really any content in any channel in Slack. All the services we mentioned above that publish webhooks could be integrated with Slack - for instance, we could get notified on Slack of a ticket opened in Jira via webhooks.

Identity management

We can talk about security with Okta offers event hooks. Okta notifies subscribers about anything that happens on their platform – new user subscription, user activation and deactivation, password change, application addition to a user’s profile, and much more. This guide covers Okta’s event hooks and how they work.

Other use cases

Collaboration (Atlassian), customer service (Intercom), email delivery (Mailchimp), payment processing (Stripe)... the list goes on. You can expect to see webhooks for platforms that need to offer information about events as they happen, either in subscribing (incoming) or publishing (outgoing) form.

Handling webhook spikes

To handle spikes of incoming webhooks, for example on a holiday where traffic patterns differ from usual, you can implement a queuing system that defers the processing of webhooks payload until a time that you have workers available

Error handling of webhooks

If the publisher does not support retry for requests that fail, or if you miss all of the retries due to an error, the only way to fix this problem is to create a separate ingest point that never fails, log all the data that comes through it, and then forward and retry those requests that fail.

Debugging webhooks

To debug webhook calls that fail on your end, you can log the payload data and assign an ID to each request so that you can later correlate those requests with your internal data, or if that’s not enough, you can develop tooling to analyze webhook payload.

Fanning out webhooks

To get your request in two different places (like a log and an actual action, for instance), you can implement a fan-out pattern to get the request and redirect it to two or three other endpoints. Of course, you’d have to keep track of which endpoint succeeded and which one failed and apply a retry logic.

There are solutions to all these problems, though they take some work to get right.