Taking Advantage of Increasingly Popular Webhooks

Profile Picture

Maurice Kherlakian

Thu May 06 2021

Historically, if you as a developer wanted to get real-time information about an occurrence that exposed an API in your system, you would have to poll the API. You would then have to watch and wait for a status change or a 2XX response. It wasn't until 2007 that the term "webhook" was coined by Jeff Lindsay, and a new alternative to polling was born. Instead of the arduous task of manually collecting information, we can now ask the server that controls the resources we’re interested in to notify us whenever a change occurs. Essentially, webhooks are the computer version of the famous Hollywood adage “Don’t call us, we’ll call you!"

This article covers the benefits of webhooks, integration webhook publishers (API platforms), some of the problems related to their utilization, and how to solve them.

Benefits of using webhooks

Eliminates polling

Webhooks were not well known when the term first came on the scene, but have become more popular because of the many benefits they bring. Primarily, webhooks eliminate the need for polling, a task that is painful for both the provider and the consumer. For the provider, polling creates overhead (the client has to request the resource many times before getting the response that they expect). For the consumer, polling requires complex code with retry logic and rate-limiting that engineering teams must maintain.

Simple HTTP request

Webhooks are extremely simple to use and usually come in the form of a straightforward HTTP request initiated by the publisher. There is no dependency on any language or other technology. The HTTP method you will receive the webhook with is generally a POST, and the payload is generally JSON or XML, though it could be any text-based format. Part of what makes webhooks so attractive is that if you’re developing an API, all you have to do is add one endpoint to process incoming webhook requests, as you already have all the infrastructure.

Example of consuming a webhook

Let’s say that you want to receive events related to Shopify on

https://www.myawesomesystem.com/shopify.

After configuring your URL in the publisher’s platform (in this case Shopify), if you’re using node.js and express, you can write:

const express = require('express');
const app = express();
app.use(express.json());
app.post('/test', (req, res) => {
   console.log(req.body);
   console.log(req.body.test);
   res.sendStatus(200);
});
app.listen(8000, () => {
   console.log('listening on 8000');
})

Notice here that we’re using res.sendStatus(200), which is equivalent to res.status(200).send(“OK”). Express 4.x introduced that syntax.

We’ve omitted some details to keep this article simple. However, some of them are worth noting:

  1. Shopify verifies the consumer’s SSL certificate and won’t send the payload if the verification fails.
  2. Shopify will also send a few request headers along with the HTTP request. They contain the Shopify topic, a sha256 signature so that you can verify the payload, and a couple of other helpful tidbits of information.

API providers who publish webhooks

Now that we have a good idea of how a webhook is consumed, let's go over who publishes them. Platforms that need to offer information about events in real-time will often offer webhooks. Generally, any system that needs to expose state information can do so with a webhook.

Ecommerce

Shopify, for instance, offers webhooks that allow us to know all sorts of events that happen on their platform. Two commonly implemented use cases are inventory synchronization and order fulfillment – Shopify sends a webhook to subscribers with a payload containing information about the purchase. There are many other use cases, for example we have an article on synchronizing your Shopify orders with webhooks.

How to Synchronize Orders with Shopify Webhooks

Learn how use the order/created, order/updated, and order/deleted shopify webhooks to synchronize orders on Elasticsearch.

Sat Feb 20 2021

CI/CD

Github also offers webhooks. Perhaps the most common use case is to get notified when a push occurs on a branch. Those events allow subscribers to start a CI pipeline that builds our application and deploys it.

Communication

Twillio is known for being developer-friendly. Their webhook offering helps create integration based on SMS triggers. There are many other use cases such as automating Vercel Deploys via SMS using Twilio webhooks or building a customer rating app with Twilio webhooks.

Slack also offers the ability to post messages to channels using webhooks that you can create from the platform's UI or its API. This allows you to post really any content in any channel in Slack. All the services we mentioned above that publish webhooks could be integrated with Slack - for instance, we could get notified on Slack of a ticket opened in Jira via webhooks.

How to Automate Vercel Deploys with Twilio SMS webhooks

Learn how to automate vercel deploys everytime your Twilio number receives a message.

Sun Apr 18 2021

How to Build a Product Rating Service with Twilio SMS Webhooks

Learn step-by-step how build a product rating service by leveraging Twilio SMS webhooks.

Tue May 04 2021

Identity management

We can talk about security with Okta offers event hooks. Okta notifies subscribers about anything that happens on their platform – new user subscription, user activation and deactivation, password change, application addition to a user’s profile, and much more. This guide covers Okta’s event hooks and how they work.

Other use cases

Collaboration (Atlassian), customer service (Intercom), email delivery (Mailchimp), payment processing (Stripe)... the list goes on. You can expect to see webhooks for platforms that need to offer information about events as they happen, either in subscribing (incoming) or publishing (outgoing) form.

Handling webhook spikes

To handle spikes of incoming webhooks, for example on a holiday where traffic patterns differ from usual, you can implement a queuing system that defers the processing of webhooks payload until a time that you have workers available

Error handling of webhooks

If the publisher does not support retry for requests that fail, or if you miss all of the retries due to an error, the only way to fix this problem is to create a separate ingest point that never fails, log all the data that comes through it, and then forward and retry those requests that fail.

Why Implement Asynchronous Processing of Webhooks

Learn about implementing asynchronous processing to handle volume spikes in a timely and efficient way.

Wed May 05 2021

Debugging webhooks

To debug webhook calls that fail on your end, you can log the payload data and assign an ID to each request so that you can later correlate those requests with your internal data, or if that’s not enough, you can develop tooling to analyze webhook payload.

Complete Guide to Debugging Webhooks

Learn what brings bugs in the process of developing with webhooks, how to tackle these bugs, and tools that can help the debugging process.

Sun Jun 27 2021

Fanning out webhooks

To get your request in two different places (like a log and an actual action, for instance), you can implement a fan-out pattern to get the request and redirect it to two or three other endpoints. Of course, you’d have to keep track of which endpoint succeeded and which one failed and apply a retry logic.

There are solutions to all these problems, though they take some work to get right.

How to Fan Out Webhooks to Different Services

Learn how to use Hookdeck to fan out webhooks to different services.

Wed May 05 2021

Conclusion

This article and others in this series were hopefully helpful in understanding how webhooks work and how to build a resilient infrastructure to handle them. Webhooks are, in most cases, an integral infrastructural piece that might lead to losses in case of service interruption. It is therefore essential to build resilience into your projects.

Hookdeck is a feature-complete webhooks handling platform that handles resilience and monitoring for you. It enables you to decide how to route webhooks requests, debug and visualize your payloads, offer insight into errors and trends, retry on delivery failure, hook into your development process with a local proxy, and much more. Try Hookdeck now for free by creating an account here.

Try Hookdeck Today

Instantly & reliably manage
your webhooks

Learn More