Guide To CircleCI Webhooks Features And Best Practices

Now that you have gotten all the information you need to get started with CircleCI webhooks, examined the different types of use cases for the technology, and implemented CircleCI webhooks with a simple tutorial, now let's look at what you should consider when moving your webhooks to a production environment.

Development environments are very predictable, and the conditions under which your webhooks operate are controlled. However, in a production environment, issues related to system resources, security, response latency, and other things can arise. For guaranteed uptime of your webhooks, you need to make sure that you're taking the necessary preventive and corrective steps.

CircleCI Webhooks Features

Best Practices for Working with CircleCI Webhooks

Local troubleshooting before deployment

The first box you want to check is local troubleshooting. You need to ensure that you have properly tested your webhooks to make sure that the following operations are taking place:

• The webhooks are being received at the webhook URL.
• There are no 404 errors, i.e. the webhook URL can be found.
• If you're authenticating your webhooks, ensure that they pass the authentication checks.
• Assert that your webhooks have the desired effect on your external client.

Webhooks require a live, publicly accessible HTTPS URL to function which makes them tedious to troubleshoot in local development environments. This is why you don't want to debug errors in production, as this can have a very damaging impact on your business.

To overcome the frustrations of debugging, the Hookdeck CLI is available to help you generate a webhook URL that tunnels your webhooks to your local development server. You can also examine your webhook payload and inspect the request headers.

Asynchronous webhook processing

CircleCI has a five (5) second timeout for each webhook request sent. This means that if CircleCI does not receive a response within this time, it will assume that the delivery has failed and retry the webhook at a later time.

If you will be receiving a lot of webhooks, you want to ensure that you're processing them as fast as you can and returning a response within CircleCI's timeout range. Processing speed is often limited, but you can ensure that you're returning a response to CircleCI as fast as possible by processing your webhooks asynchronously.

Asynchronous processing can be introduced into your infrastructure through the use of message queues like Amazon SQS or RabbitMQ. A message queue helps to quickly ingest your webhooks from CircleCI and return a response immediately. You can then serve your webhooks to your destination client at a rate that it can handle, and avoid server crashes due to traffic spikes.

CircleCI webhook security using webhook signatures

Security is of the utmost importance in a production environment. Because your webhook URL is publicly accessible, any person or web agent can send a request to it. Attackers can craft a malicious web request in order to inject a harmful payload into your system. This is why you need to verify that the source of your webhook requests is CircleCI.

Fortunately, CircleCI webhooks provide a way to achieve this. When you add a secret token to your CircleCI webhook form, CircleCI cryptographically encrypts the payload and puts the signature in the circleci-signature header.

This way, you can use the same secret token on your client to verify that your webhook is indeed from CircleCI.

In our CircleCI webhook tutorial, we demonstrated how this can be done using the HMAC algorithm. You can also find more information about this on CircleCI's webhook documentation page.

Server IP whitelisting

Another way to ensure that your webhooks are originating from CircleCI is through IP whitelisting. You can configure rules on your server that uses an allow list of IP addresses to ensure that only requests from CircleCI's servers are accepted by your endpoint.

Every other request from unknown sources will be automatically bounced by your server and will not make it to your endpoint.

Retrying failed webhooks

CircleCI treats every webhook that returns a non-2xx HTTP status code as a failed webhook. CircleCI webhooks also have intelligence built in to try these failed webhooks at a later time.

However, for guaranteed self-healing of failed webhooks, you might also want to mitigate this issue with a custom retry system just in case CircleCI does not retry your failed webhook.

Retry systems can be simple cron jobs, but the most effective ones keep a repository of failed events to be retried with their payload, and also use intelligent timers to trigger the retries. You can also combine your retry system to your message queue when processing your webhooks asynchronously.

Retrying failed webhooks ensures that data integrity issues do not arise in your API. Failed webhooks in commercial or financial systems can lead to users not getting value for goods and services paid for, thus you want to ensure that you're not dropping any webhooks.

Logging and monitoring CircleCI webhooks

To ease troubleshooting your CircleCI webhooks in production, a logging system is highly recommended. You need to be able to trace your webhook requests and be aware of their status at all times, especially in situations where they fail.

When webhooks fail, information about their failure helps in debugging and fixing them. You can perform logging using different types of strategies, from writing logs to simple flat files to using a standard logging service.

Idempotency for webhook duplicates

A very important check to implement when working with CircleCI webhooks is ensuring that a single webhook is not processed more than once. Oftentimes, CircleCI can send a webhook request more than once and you need to ensure that you're not duplicating the impact of the webhook on your destination application.

Duplication of webhook processing can quickly compromise your data and make your application lose its integrity. To help with this, CircleCI sends an id property in the webhook payload. This id can be saved to the database when a webhook is received and compared against new webhooks to avoid duplication.

Conclusion

Following best practices allows you to benefit from the wealth of experience professional developers have to offer, and avoid mistakes that could cost you a lot of business. In this article, we have looked at best practices you should consider taking when deploying your CircleCI webhooks to production environments.

A good number of these best practices can take a significant amount of time and expertise to set up properly. If you want to build fast and ensure that you're taking into consideration best practices for your webhooks, give Hookdeck a try today.

Hookdeck has in its suite tools like a retry configuration system, a rate limiter, queueing service for asynchronous processing, and an intuitive dashboard that provides clear visibility into your GitHub webhooks.

Happy coding!