Overcoming Event-Driven Architecture Complexity with an Event Gateway

EDA seems like a great idea, offering a mixture of flexibility and scalability that isn't always possible with traditional application architectures. You embark on the journey, only to find out that things are starting to get complex quickly. Message receivers aren't performing well because they have to filter through messages on their own. Third-party application orchestration is taking up too much of your precious developer time. Customers are asking for webhooks to support easier integration but you don't have the necessary infrastructure to support it.

Event gateways can help you overcome many of these challenges as your EDA becomes more complex over time. Let's look at some common scenarios that increase the complexity of your EDA and how we can overcome them with an event gateway.

Filter messages before they are dispatched

When first starting out with an event-driven architecture, you may have a small number of messages exchanged between the message publisher and your receivers. Dispatching seems almost instantaneous and performance isn't an issue. You may even add more receivers to do different things based on a specific set of messages.

The code for your message receiver may wish to only process messages with a specific value in the message body, or only if it matches a specific condition. While this is an easy fix in code, it means that your message receiver may end up handling hundreds or thousands of messages that it will simply ignore. This can make your receiver inefficient and perhaps even struggle to keep up with the number of messages received as your application grows.

As the number of messages being processed increases, it is important to only send messages to their destinations when they are needed. Filters allow your message broker to make a quick decision: “should I send this message to the receiver?” By suppressing the delivery of messages that are not interesting to your receiver, you free up precious compute resources to focus on processing only the messages of interest to your receiver.

In some cases, this may yield cost savings as well when using a serverless architecture that would be required to spin up a new receiver, only to find out that it wasn't a message of interest.

Evaluate your likely use cases and determine if message filtering would be of benefit to your EDA. While you may not need filters on day 1 of your production environment, filtering is often a requirement for day 2 operations to manage both cost and performance of your application.

Add routing rules to reduce code complexity

As the number of integrations grows, it may be necessary to evaluate a received message and determine where it should be sent. The typical approach for developers is to write some code that:

1. Evaluates the incoming message to determine where it should go
2. Compose a new message or clone the incoming message
3. Publish the new message to a new channel

While this may seem like a straightforward approach, more integrations will require more dedicated services whose job it is to route messages.

To avoid writing this routing logic yourself, look for routing rule capabilities in your event gateway. This will allow you to send a message to a specific application or duplicate the message and send it to multiple applications at the same time.

Routing rules may also be used to dispatch messages to third-party applications, acting as an orchestration layer without the need to write, deploy, and manage a dedicated service.

Transform messages for faster integration

Sometimes messages just aren't in the correct format. This may be due to limitations from third-party applications or the need to support different industry standards. In this case, transformation support in your event gateway enables you to convert a message into a new structure.

Perhaps your transformation needs to transform a JSON message body into a slightly different JSON format due to some versioning changes and the need to support backward compatibility. Or, perhaps you need to integrate with a third-party application that only supports XML and you prefer to transform their messages into a JSON format that your internal applications can understand.

In some cases, transformations may be combined with filters to only transform or filter messages that meet specific criteria. For example, a transformation may apply more complex logic based on event properties and then add a custom property that is used by a filter. Or a transformation may only be applied if a filter condition is met.

Subscribe to third-party events to orchestrate applications

Orchestrating multiple third-party applications, often via APIs and webhooks, requires lots of time to write the code necessary to work with both applications. It takes considerable time to write each integration, test it, deploy and maintain it, and upgrade it when something changes.

Transformations can also be a powerful tool to help you avoid writing any orchestration complex code. Orchestrating multiple third-party applications may only require you to register to receive a message from one app, perform a transformation of the message to a new format, and dispatch the transformed message to a second application.

Another advantage of leveraging an event gateway to orchestrate third-party applications is that they often offer accelerated creation. Once the event gateway has received a message from a third party, it can be used as a template to construct your transformation, routing, and filtering logic. This enables the developer to work with live data, proceeding with the effort in confidence rather than guessing what the message will look like based on hand-written and error-prone documentation.

Extend the reach of your events by supporting webhooks

As your application expands, there are likely emerging opportunities to integrate with other ecosystems. Perhaps your application is an email campaign manager that manages drip campaigns. You have received requests from multiple customers who want to know when a step in the drip campaign has been completed. You determine that offering webhooks for your events is a good idea.

You start to look at what it will take to build your own webhook dispatcher to notify your customers when an event occurs. While it seemed like a simple task to make a POST request to their URL of choice to receive the events, you start to realize that you need to add in failover support should their webhook receiver fail or be offline. You realize that you need a two-way handshake to ensure that they aren't processing messages from an unknown application. Mid-way through development, you realize that you could end up generating lots of messages all at once, so you need to add in batching and rate limiting to prevent overwhelming your webhook subscribers. Oh, and you forgot to add some REST APIs to manage subscriptions for your customers' applications. Pretty soon, what started as a POST request to a URL is a full-time job of building and maintaining a webhook dispatcher product!

Event gateways can take all of this off your plate. Instead of building your own webhook dispatcher from scratch, use an event gateway to manage everything for you. You push a message to the event gateway and it does all of the work of dispatching the message to registered webhook URLs, implementing authorization, implementing rate limiting, and handling errors. Plus, most event gateways support viewing recent messages to help you troubleshoot developer issues.

Encourage event governance through reporting

As your EDA grows, you will have a variety of message formats and schema definitions. Consider leveraging your event gateway's capabilities to track and report on messages, including how they are used, what applications use them, available message channels where messages are dispatched, and any filters or transformations assigned to each channel. This will encourage a “consume when possible, build when necessary” culture of message reuse.

Final thoughts on EDA complexity

Often, EDA complexity enters into applications seemingly overnight. Everything is manageable until it reaches a tipping point and requires the capabilities of an event gateway. Therefore, starting with an event gateway early will prepare you for when your EDA grows and becomes more complex than what code alone can manage. Consider investing in an event gateway from the start and leverage features only as you need them to manage your EDA complexity.