architecture | Jun 2, 2021
Hookdeck vs RabbitMQ: Which way to go with your webhooks?
Fikayo Adepoju Oreoluwa
Asynchronous processing is highly recommended in modern architectures as it helps build highly scalable applications that can handle high traffic from client requests. However, with this great power comes the huge responsibility of maintaining a more complex system than one that processes requests synchronously. One of the very common ways to deal with increased complexity in system architectures is to take advantage of third-party services (buying as opposed to building). These third-party services offer architectural components, that you would previously have had to build on your own, as a service (IaaS).
In this article, we will take a look at the asynchronous processing of webhooks using message queues. We will compare the building approach to processing webhooks, using RabbitMQ, with the buying approach using Hookdeck.
Introduction to Asynchronous Processing
Read an introduction to asynchronous processing. We compare the differences between synchronous and asynchronous and what are the alternative solutions to handling asynchronous processing.
Introduction to Messages Queues
Learn what are message queues, what are producers and consumers, the advantages of using message queues and what frameworks and services you can use to implement message queueing.
What is RabbitMQ and How Does It Work?
RabbitMQ is a lightweight message broker open-source framework used for building message queues for asynchronous processing. RabbitMQ is one of the most popular message broker frameworks and can be deployed on-premise or in the cloud.
Just like other message brokers, RabbitMQ facilitates the communication between message producers and consumers. However, instead of using topics or a named queue for routing messages, RabbitMQ introduces the concept of an exchange. Exchanges are message routing agents that exist within RabbitMQ itself.
Producers publish messages to an exchange, and RabbitMQ applies routing rules and sends copies of the message to all applicable queues.
What is Hookdeck and How Does It Work?
Hookdeck is an Infrastructure as a Service System for processing webhooks. Hookdeck provides a message queue that asynchronously processes webhooks by ingesting webhook requests from your SaaSapplications, and distributing them to your callback endpoint based on the load your API can handle.
Hookdeck also comes with a logging system for your requests, as well as request filtering, retries, and a dashboard to search and troubleshoot your webhooks.
Processing Webhooks: Rabbit MQ vs Hookdeck
In order to get first-hand experience with the two technologies, I decided to process webhooks from my Shopify store to my backend API using both RabbitMQ and Hookdeck. I went through each process and compared both services on their requirements, setup process, reporting, and maintenance.
1) Requirements
RabbitMQ
To set up asynchronous processing of webhooks using RabbitMQ, I needed the following:
- Knowledge of RabbitMQ supported communication protocols (AMQP or STOMP)
- Knowledge of one of the RabbitMQ supported programming languages (
PHP,JavaScript,Go, etc.) - Proxy server to receive the webhook requests and publish messages into RabbitMQ (HTTPS endpoint required for receiving webhooks)
- RabbitMQ setup
- Hosting service to host the proxy server and RabbitMQ (to be hosted on different machines)
- An HTTPS endpoint to my backend API
Hookdeck
To achieve the same webhook routing using Hookdeck requires the following component(s):
- An HTTPS endpoint to my backend API
2) Setup Process
Here, I am going to list the number of steps it took to get asynchronous processing working with both technologies. My setup is simple as there are no advanced routing logic or horizontal scaling techniques involved, just simple publishing, queuing, and consumption of messages.
RabbitMQ
- Install and run RabbitMQ on Heroku using CloudAMQP
- Create proxy message publishing server using Node.js and the amqplib package
- Deploy proxy server to Heroku
- Replace Shopify webhook endpoint with that of the proxy server
- Write logic in my backend API to consume messages from the RabbitMQ using the
amqplibpackage
I have hidden sub-steps from this list to keep it clean (some of these steps contain up to 5 or 6 sub-steps).
Hookdeck
Here is a list of steps I took to successfully get my Shopify webhooks to be processed through Hookdeck's message queues.
- Create a new connection
- Name your connection (for me this was
Shopify Store Hooks) - Enter destination label (for me this was
My production API) - Enter destination URL (your backend
httpsendpoint) - Deploy connection (click the
Create Connection) - Replace the endpoint in Shopify with the one generated by Hookdeck after the connection has been created
Unlike the steps listed for RabbitMQ, I have included the sub-steps here and this is all there is to it. The entire process took about 5 minutes, including testing out the setup.
3) Reporting
Having visibility into your webhooks processing is very important in this setup. So what do the two systems have to offer?
RabbitMQ
RabbitMQ has a management plugin that provides you with an HTTP-based API for management and monitoring of RabbitMQ nodes and clusters, along with a browser-based UI and a command line tool, rabbitmqadmin.
However, the RabbitMQ management console only gives you information about the queue and not the entire process. You will need different monitoring tools to track the request from your webhooks provider to your backend API.
The console also cannot set alarms for certain events you will be interested in, for example, errors or performance issues within the system.
Hookdeck
Hookdeck has a dashboard built for tracking and analyzing webhook requests. Every single bit of information regarding your webhook request is captured and accessible to you. Hookdeck also adds metadata like request timestamps, the status of your requests, and how many times the request has been attempted. The dashboard helps you make sense of all captured information by visualizing your data in an easily comprehensible way.
You can also set up alerts to be notified when something important happens so that you can take action promptly.
The Hookdeck dashboard also allows you to retry failed webhook requests, configure automatic retries and search webhook events.
4) Maintenance
It's one thing to successfully get a setup running, but it's another thing to keep it running without issues. Maintenance costs in terms of money, human resources, and time investment all play a huge role in choosing which setup to adopt.
Let's take a look at the cost of maintaining the two approaches to processing our webhooks.
RabbitMQ
One of RabbitMQ's most attractive features is that it is very flexible, and routes can be dynamically configured. The RabbitMQ approach gives you total control over your entire setup and you can configure it to suit the requirements of your application.
With this flexibility comes a very huge maintenance responsibility. You need experts with a very good understanding of how RabbitMQ works and how it can be configured. Also, you need to synchronize the right versions of RabbitMQ and AMQP library with your operating system and update these any time the hosting environment is updated or you move to a completely different host.
There is also the cost of running host servers and the responsibility of managing and scaling them.
Hookdeck
Being an IaaS, Hookdeck is fully managed by the company behind it. You don't need to worry about scaling servers, security patches, software updates, and so on. You also don't need expertise in message queues to run and maintain a fully functional one.
However, with this hand-holding comes a compromise over the amount of control you have over the setup. In most common use cases this would not matter, but there may be rare cases where you would need something customized; the ability to do this is not currently available.
Conclusion
In this article, we looked at two approaches to processing webhooks. The RabbitMQ approach gives us the ability to build from scratch and configure the entire process of handling our webhook requests. The Hookdeck approach helps to quickly set up asynchronous processing of webhooks using Hookdeck's battle-tested suite.
One thing is clear: if you need to set up message queues for your webhooks fast and easily, have full-fledged monitoring and alerting tools, search through webhook events and configure automatic retries for failed requests, then Hookdeck is the right approach.
However, if your infrastructure demands require full control of the setup, hosting platform, and software installations, then it's better you invest in rolling your own message broker setup using RabbitMQ.
Related Guides
Business Case: Hookdeck vs Amazon EventBridge
Read our business case to compare Hookdeck and Amazon Eventbridge for webhook processing based on: architecture, complexity, costs, security, scalability, and reliability.
Why Implement Asynchronous Processing of Webhooks
Learn about implementing asynchronous processing to handle volume spikes in a timely and efficient way.
Best Practices When Deploying Webhooks in Production
In this article, learn what are the practices for setting up , scaling and error recovery to deploy webhooks in production.