Hookdeck: The Modern Queueing Solution for Your Webhooks
In the previous article, we took a deep dive into the type of attributes that a dependable webhook solution should possess. We also looked at features that will improve the experience of teams working with webhooks. We then looked at the drawbacks of building the solution from scratch and introduced Hookdeck.
In this article I will discuss the impact Hookdeck reliably handling your webhooks has on your workflow, including how it reduces your time to value on webhook integrations.
Before jumping in, if you’d like to know more about how Hookdeck ensures you never miss a webhook, read our blog post on Hookdeck’s commitment to reliability which covers our architecture and organization best practices here.
Hookdeck’s webhook resilience out of the box
Let’s jump in and discuss the resilience you get without touching a single line of code when you manage webhooks with Hookdeck.
- Rate limiting: You can adjust the request rate to your consumers to serve them webhooks at their own pace. This helps to avoid overwhelming your server capacity.
- Automatic Retry: Hookdeck automatically retries failed webhooks. This way, you can self-heal from transient faults within your webhook pipeline.
- Manual Retries: Some failures require you to troubleshoot the system and make updates to fix the issue. Webhooks that fail while this type of fault persists can be manually retried from your Hookdeck dashboard to reconcile them with your application.
- Issues: When a problem results in failed webhooks, for example a server error 500 or client error, Hookdeck automatically creates an Issue to track the problem. You can then browse the issue created to view information like the server response, payload, timing information, etc. The issue created also collates all the webhooks that failed due to the problem, so you can resend them with one click once the problem is fixed.
- Issues Alert Triggers: You can also set up notifications to be alerted when an issue is created or its status is updated. These alerts allow you to stay on top of the situation when a problem is active and take steps promptly to fix the issue before it impacts your users.
- Bulk Retry: It is common for multiple webhooks to fail when a failure persists for a while. After fixing the issue, it is counter-productive and time-consuming to retry these failed webhooks one at a time. Thus, Hookdeck gives you the ability to perform a bulk retry. A bulk retry allows you to retry all your failed webhooks at once. And because we have rate-limiting set up, you can rest assured that your servers will not be overwhelmed.
Hookdeck’s observability out of the box
Throughout this series, we have emphasized that a reliable webhook solution's two core traits are resilience and observability. Let's now look at how Hookdeck ensures that observability is also equally present.
Hookdeck comes with webhook-tailored observability out of the box. You can see a complete trace of your webhook request from your webhook provider, through Hookdeck, to your destination server. Below is a breakdown of how Hookdeck approaches the observability and monitoring of webhooks.
- Complete visibility of the lifecycle of your webhook from provider to destination.
- You can view and inspect the raw webhook properties from your provider and do the same for the request sent from Hookdeck to your server.
- Browsable webhook payload and headers. Requests status and timestamps are also made available.
- You can view defined status pages like Successful, Failed, Pending, and Paused. You can also configure custom pages based on specific webhook behaviors/issues/states.
- Historical view of webhook patterns (success/failure) using defined time ranges or a custom range.
- Ability to configure alerts for your most critical issues and webhook failures.
Fastest time to value on your most common webhook workflows
How Hookdeck is solving your most burning pain points
As a developer/team working with webhooks, your sole aim is to release an integration. Whether it is Stripe, Shopify, or GitHub, your task is to integrate the SaaS application with your application.
With Hookdeck, you can focus on that single task without worrying about building the infrastructure to support it. Hookdeck achieves this by allowing you to set up your integration in a matter of minutes.
All you have to do to have an integration up and running is to create a Hookdeck Connection between your webhook provider and your destination server, and that's it. Within the time it took you to read the last paragraph, you could already have a webhook connection set up on Hookdeck. The time-to-value on integrations is second to none.
And not just on integrations! Hookdeck also reduces time to value on other everyday webhook management tasks. Our ”Webhook Problems and Solutions” series shows you how all the routine tasks of managing, testing/troubleshooting, deploying, and monitoring webhooks have been highly optimized.
In the next section, we will look at some of these common tasks in order to see how Hookdeck gives you the best time to value while improving the developer experience.
Testing webhooks locally
Task: I want to send a webhook locally and replay it.
Receiving webhooks locally before and after Hookdeck
| Before Hookdeck | After Hookdeck | |
|---|---|---|
| Estimated time | 6x faster | |
| Step 1 | Set up a local webhook URL with a tool like ngrok (you need to generate one each time you want to test an integration) | Connect the Hookdeck CLI to your Hookdeck connection (each integration has a dedicated webhook URL, this does not need to change each time you need to test locally) |
| Step 2 | Submit the URL to your provider (you need to submit a new one each time you set up the test environment) | Submit your Hookdeck-generated webhook URL to your provider (this is a permanent setup. You don’t need to change the webhook URL next time you need to test the integration) |
| Step 3 | Receive webhooks locally | Receive your webhooks locally |
| Step 4 | Generate a new event each time you need to replay | Replay a webhook as many times as you want from the webhook event page |
Let’s assume a single issue takes an average of 3 troubleshooting cycles and 5 replays per cycle in order to resolve the issue.
1 Troubleshooting cycle requires generating a new URL and submitting it to provider = 5mins
1 replay requires generating a new event to mimic the error event = 3mins
Keeping all other activities constant:
Without Hookdeck
Total time spent on generating new URLs = 3 x 5mins = 15mins
Total time spent generating new events = 3 x 5replays x 3mins = 45mins
Total time spent on routine troubleshooting tasks = 15 + 45 = 60mins
With Hookdeck
Total time spent on generating new URLs = 1 x 5mins = 5mins
Total time spent generating new events = 1 x 3mins = 3mins
Total time spent on routine troubleshooting tasks = 5 + 3 = 8mins
Time saved fixing an issue using Hookdeck = 60 - 8 = 52mins
Let’s assume troubleshooting tasks are performed an average of 10 times each week.
Total time saved per week = 10 x 52 = 520mins (approx. 8hrs 40mins)
That’s more than a full work day’s worth of time saved in a week.
This same time-saving effect applies to the tasks described below as well as other common webhook workflows not discussed here.
Monitoring webhooks
Task: I want to see the trend of my webhook.
Setting up monitoring for webhooks before and after Hookdeck
| Before Hookdeck | After Hookdeck | |
|---|---|---|
| Estimated time | 5 hours - 3 days (based on expertise) | 0 mins (already available) |
| Step 1 | Define the metrics that are most important | Your webhook trends are already available in a high-level to drill-down format on the Hookdeck dashboard |
| Step 2 | Instrument your server with a metrics collection agent from technologies like Prometheus, Nagios, etc., and log collection technologies like ELK | |
| Step 3 | Set up a metrics collection database like Graphite | |
| Step 4 | Set up a visualization dashboard tool like Grafana to view your metrics |
Troubleshooting webhooks
Task: I want to see all my failed webhooks and replay them after the fix.
Searching and replaying failed webhooks before and after Hookdeck
| Before Hookdeck | After Hookdeck | |
|---|---|---|
| Estimated time | Setup 1-2 days, Search/Replay: 30 mins -1 hour | No setup required, Search/Replay: avg. 3 mins |
| Step 1 | Collect logs with technologies like ELK stack. Ensure that the log data contains complete information for replaying the webhook and also capture the webhook status | Click the Failed status page under the Event menu |
| Step 2 | Store your logs in reliable storage with querying tools like Big query or Logstash | Use the bulk retry dropdown to retry all failed requests asynchronously |
| Step 3 | Create a visualization dashboard or use a CLI tool to query your logs for failed requests | |
| Step 4 | Collate all failed requests | |
| Step 5 | Create an automation script that takes all the failed requests and retries them |
And there’s more….
Hookdeck is a single-pane-of-glass management that allows you to manage all your webhook activities in one place and onboard team members quicker by giving them one tool to learn instead of multiple.
Aside from the core resiliency and observability tools that Hookdeck already provides, it also consists of additional tools that improve your experience with webhooks and provide more customizations.
Some of these features are listed below.
- Signature verification allows you to set up HMAC signature verification between your webhook providers and their destination(s).
- Transformations allow you to modify your payload to suit an expected structural or data format before it hits your server.
- Filters allow you to apply conditional rules to the webhooks you allow based on the path, payload, or headers.
- Bookmarks allow you to pin a webhook for future use, like retrying or inspecting it at a later time.
Hookdeck centralizes all these features in one place unlike other solutions where you have to jump across different dashboards, management tools and environments to complete a workflow (for example monitoring can take you through Elastisearch or Grafana just to track a webhook trace).
The centralized management also helps to harmonize webhook workflows from different providers. Currently, webhooks from various providers, though similar, still possess some specificities in how they operate. Hookdeck understands this and is built to help you align each webhook to a single workflow pattern while providing tools to accommodate each provider's peculiarities.
Conclusion
Hookdeck's webhook-focused approach makes it designed to alleviate the pain points teams have faced working with webhooks. This mindset has inspired several reliability features, as well as features that improve the all-around developer experience.
Get started with Hookdeck today for free.
Gain control over your webhooks
Try Hookdeck to handle your webhook security, observability, queuing, routing, and error recovery.