Build Asynchronous APIs with Hookdeck

If you're building an application that relies on events from your customers, devices, or SDKs, you need a reliable and scalable way to handle those events. Hookdeck provides a robust infrastructure for building asynchronous APIs that can efficiently process and route events from various sources.

Asynchronous APIs are API endpoints whose requests are not immediately processed. They are often used to handle events that kick off long-running processes or that do not require an immediate response. This approach allows for better scalability and responsiveness in systems dealing with high volumes, or unpredictable number of events and time-consuming operations.

Skip to the Quickstart ->

Read the quickstart guide for a practical example on how to get started.

This guide provides an overview of how to use Hookdeck as an asynchronous API Gateway when building asynchronous APIs. It assumes that you are receiving events from customers, devices, or SDKs, and that your application will process these events asynchronously.

This guide covers a core subset of potential functionality:

1. Configuring your Sources, Destinations, and Connections
2. Setting up rules to retry, filter, and route events
3. Receiving events, authentication, and verification
4. Testing your application on a local server
5. Setting a custom domain
6. Error handling, alerting, and monitoring
7. Best practices

Understanding Sources, Destinations, and Connections

Hookdeck acts as an intermediary between event producers and your application. To receive and process events, you need to define a Source , a Destination and configure a Connection between them.

Sources as API endpoints

A Source represents an entry point for events in your asynchronous API. For each Source, you will be provided with a unique URL that acts as an API endpoint. Typically, Sources will be named after the client, device type, or SDK version that sends the events.

Sources can be configured with different authentication methods, allowing you to secure your API endpoints based on your requirements.

If you are receiving events from your users, create a Source per user facing endpoint. This allows you to manage access control and authentication for each endpoint individually.

Custom response

Each Source can be configured to return a custom response to the client. This feature is useful for acknowledging the receipt of an event or providing additional information to the client.

Set a custom response ->

Restrict allowed HTTP methods

You can restrict the HTTP methods that are allowed on your Source. This feature is useful for enforcing a specific method for your API endpoints, such as only allowing POST requests. By default, all HTTP methods are allowed except GET

Restrict allowed HTTP methods ->

Destinations as event consumers

A Destination represents the service that processes events in your asynchronous API. A Destination is the HTTP URL where Hookdeck will deliver the Events. Typically, the Destination is an endpoint in your application that handles event processing.

You can have a single Destination to represent your entire API, or use different Destinations for various event types or processing requirements.

Delivery rate control

Hookdeck allows you to set a maximum delivery rate for each Destination. This feature helps you control the flow of events to your processing endpoints, preventing overload and ensuring smooth operation during peak times or unexpected surges in traffic.

For example, if you set a delivery rate of 60 events per minute, Hookdeck will ensure that no more than one event per second is sent to your Destination, even if events are received at a higher rate. Events beyond this limit are queued and delivered as capacity becomes available.

The delivery rate in conjunction with multiple destinations can be used to enforce limits on the numbers of events processed for a specific endpoint. This is useful if you want to attribute different processing capacities to different users.

Set a max delivery rate ->

Connections for flow control

A Connection consists of a Source, a Destination and set of optional rules. It defines the flow of the Event , from requests through to delivery. When an event is received, Hookdeck will create a new Event for each eligible Connection.

Rules allow you to configure the behaviour of Events received on the Connection, including retries, filters, and transformations.

Retries

The retries rule lets you configure automatic retries for Events that do not receive a 2XX status code response from your Destination. You can configure the number of automatic retries and the time interval between each Attempt . You can retry for up to a week and 50 delivery Attempts. Each delivery Attempt is logged, and your Destination response data can be used to determine the cause of the failure.

Filtering

Filter rules allow you to deliver Events matching specific criteria to a Destination.

Filters can be used in conjunction with multiple Destinations to route Events to different endpoints based on their content. This can help reduce load on your backend by screening out unnecessary events early in the pipeline, ensuring that your processing resources are used efficiently.

Transformations

Transformation rules allow you to modify the payload of an event before it gets delivered to a destination. This feature can be used for:

• Normalizing data formats from different clients or versions of your API
• Converting event payloads to a common schema
• Enforcing a specific schema

For example, you could use a transformation to standardize date formats across different event sources or to add additional metadata to events before they reach your processing logic.

Receiving events

Any HTTP request received on your Source URL will be represented by a new Request within Hookdeck. You can visit the Requests section of the dashboard to see your requests. Each request will be processed and generate an Event for each connection, which will be delivered to your Destination.

Authentication and verification

Securing your API endpoints is crucial. Hookdeck supports various authentication methods, the most common being API Key Authentication and Webhook Signature Verification.

• API Key Authentication: Easily implement API key-based access control for your endpoints.
• Webhook Signature Verification: Verify the integrity and authenticity of incoming events using HMAC-based signatures. Various algorithms are supported, such as SHA-256 and SHA-1.

Hookdeck automatically handles the verification process, ensuring that only authenticated and verified requests are processed.

Developing and testing on a local server

When developing your application, you can use the Hookdeck CLI to receive Events on a local server or private network. This allows you to test your asynchronous API endpoints without deploying your application to a public server.

Custom domains

You can set a custom domain for your project, which will be used within your generated Source HTTP URLs. This is useful for branding your API endpoints with your own domain, such as api.yourcompany.com.

To set up a custom domain:

1. Visit the General tab in your project settings.
2. Scroll to the Custom Domain section and follow the instructions to configure your domain.

Monitoring and error recovery

Hookdeck provides comprehensive tools for monitoring your asynchronous API and recovering from errors:

Leveraging Issues for problem resolution

Issues allow you to track and resolve problems that occur within your application. You can set Issue Triggers to open issues when certain criteria are met, such as when an event has exhausted its automatic retries or fails to be delivered.

Issues can be used to track and resolve these problems. You'll be notified via email, Slack, or another supported notification channel when an issue opens.

Using Metrics to monitor performance

Metrics provide insight into your Requests, Events, and Attempts. They are available in the dashboard and can be exported to your observability platform, such as Datadog. Metrics include important insights for troubleshooting your server response latency, delivery rates per destination, error rates, and more.

Best practices

Process events synchronously

Hookdeck acts as your queue, allowing you to process events synchronously when received from Hookdeck. It provides a 60-second default timeout window for you to process each Event and return a response.

Implementing another event queue when receiving events from Hookdeck is redundant as it already behaves as a push queue. To control the rate your server processes requests, simply set a rate limit on your Destination.

How to Control Throughput and Queue Events to Your Destination ->

Learn how to prevent spikes and control throughput

Handle out-of-order events

There's no guarantee of the order in which Events will arrive. Your application should use its own logic to order events upon receiving them. Always use relevant data from the event payload, such as an created_at value, to ensure you don't process stale requests that might overwrite more recent updates.

Hookdeck delivers Events in the order they arrive on a best effort basis, but there's no guarantee Hookdeck receives them in order.

Implement idempotent processing

Hookdeck operates on an "at least once" delivery guarantee, meaning an Event may be delivered more than once. Ensure that your event processing logic is idempotent, meaning that processing the same event multiple times does not lead to unintended side effects.

For operations that are not inherently idempotent, you can track a unique identifier in the events, such as the x-hookdeck-eventid header, to ensure the operation is only performed once.

Limitations

Schema validation

Hookdeck does not perform schema validation on the event payloads. It is your responsibility to ensure that the event payloads conform to the expected schema in your application. You can use Transformations to enforce a specific schema before delivering the event to your Destination.

Assuming the request is a valid HTTP requests and authentication is successful, Hookdeck will return an HTTP 200 status code to the client.

Schema validation is on the roadmap, and we are working on providing this feature in the future. Contact us if you have specific requirements or feedback.