Navigating Event-Driven Architecture: A Guide to Choosing the Right Infrastructure

Event-driven architecture (EDA) is experiencing increased interest by startups as well as enterprises to build scalable, responsive, and flexible applications. EDA enables systems to react to changes and events in real-time, fostering more dynamic interactions between distributed services or simply between different systems, such as internal and external services, such as API providers. However, successfully implementing an EDA requires a careful selection of infrastructure components.

This article explores the key categories of EDA infrastructure — message brokers, distributed streaming platforms, event gateways, event meshes, Enterprise Service Buses (ESBs), and data storage for message replay, providing insights into when and why to choose each. At the end of the article, I will provide recommendations based on your solution needs.

Message Broker

Message brokers are the most commonly encountered infrastructure for event-driven systems. They facilitate asynchronous communication between different components of a system. Message brokers allow services to publish messages (events) without knowing who will consume them, and consumers can subscribe to messages of interest. They are commonly found in a microservice architecture, allowing each service to operate independently while exchanging messages and events for those interested.

RabbitMQ and ActiveMQ are popular choices, while cloud vendors offer managed alternatives such as Amazon SQS, Amazon SNS, Google Pub/Sub, and Azure Queue Storage.

Choose a message broker when you need a reliable way to decouple service interactions, enhance system scalability, or manage peak loads efficiently.

Distributed Streaming Platform

Distributed streaming platforms are a good choice when dealing with high-volume, real-time data. These platforms not only facilitate message brokering but also provide capabilities for processing and analyzing streams of data on the fly due to the built-in storage of messages that support replay capabilities.

Apache Kafka and Apache Pulsar are two popular options. Cloud vendors offer managed alternatives, including Azure Event Hubs, Amazon Kinesis, and Google Pub/Sub,

Distributed streaming platforms are ideal for scenarios requiring real-time analytics, such as monitoring user activity on a website or processing IoT sensor data. Opt for a distributed streaming platform when your application demands high-throughput, fault-tolerant processing and replaying of data streams in real time.

Event Gateway

Much like message brokers, event gateways manage and route events between producers and consumers. However, event gateways act as a mediator by providing additional capabilities such as event transformation or aggregation.

Event gateways are an emerging category and therefore are often composed progressively through the use of technologies such as Amazon EventBridge combined with Amazon API Gateway and AWS Lambda. Hookdeck is an example of a cloud vendor agnostic event gateway.

An event gateway is the right choice when you need a message broker with the added ability to filter, transform, or aggregate events between components, applications, and/or third-party Webhooks. If an event gateway seems like a good match for your needs, you may wish to explore an article that compares popular event gateways.

Event Mesh

An event mesh offers a dynamic, distributed infrastructure that allows events to flow seamlessly across different services, regions, and cloud environments. Like a service mesh for microservices, an event mesh shifts many network concerns, such as routing, load balancing, and fault tolerance, to dedicated control and data planes.

An event mesh may be used within cloud, on-premises, or hybrid environments. They are more resilient to outages and often provide more optimized message and event processing due to real time network insights. Solutions like Solace PubSub+ create an event mesh that dynamically routes events, ensuring that they are delivered from any source to any destination.

Choose an event mesh when your architecture is highly distributed and requires real-time, reliable event delivery across multiple clouds, regions, or on-premises environments, and you need the flexibility to scale and adapt to changes dynamically.

Enterprise Service Bus (ESB)

Enterprise Service Buses (ESBs) emerged in the early 2000s to provide a comprehensive integration platform designed to connect and mediate communication between the many systems within an enterprise. They blend capabilities commonly found in event gateways, such as message transformation, filtering, and routing, with business process orchestration. Vendors such as Mule ESB and IBM Integration Bus support complex integration scenarios and are often complemented by other products in their portfolio to accelerate the integration effort.

An ESB is best suited for scenarios where you need to integrate a wide variety of applications, services, and data formats, particularly in legacy systems where such integrations can be especially challenging.

Data Storage for Message Replay

When starting on your EDA journey, you may not find the need to capture all messages for later processing. Instead, you may focus simply on decoupling components from traditional synchronous calls via REST-based APIs. Over time, however, you will find that the need for system recovery, auditing, and analytics will require storing some or all historical messages.

Technologies like Apache Kafka offer built-in log-based storage that drives its distributed streaming platform. Secondary data stores may also be used, configuring all messages to be stored in EventStoreDB for event sourcing solutions or InfluxDB for time-series data. These storage solutions are essential when your system requires durable storage of events for replaying purposes, whether for debugging, historical analysis, or system recovery.

Alternatively, consider an event gateway vendor such as Hookdeck that offers hosted, durable messaging for your EDA solution. This avoids the need to install, configure, monitor, and manage multiple infrastructure components by combining them into a single offering ready to scale as your needs grow.

How Do You Choose the Right EDA Infrastructure?

Choosing the right infrastructure for your EDA depends on several factors, including the nature of the events, system requirements, and specific use cases. Here are some guidelines:

• For simple, decoupled communication between services, a message broker might suffice. However, keep in mind that most message brokers lack routing, filtering, transformation, and local storage for replay.

• When dealing with high-volume, real-time data streams, a distributed streaming platform offers the necessary throughput and processing capabilities. These platforms come with increased complexity of management and configuration, so careful consideration should be taken before embarking on managing your own streaming infrastructure. Cloud vendors offer managed alternatives, but monthly costs can increase quickly.

• Consider an event gateway for message broker capabilities that need API Gateway type authentication and to manage complex event transformations or routing logic in a centralized manner between internal and external services.

• In highly distributed, dynamic environments where services need to communicate across boundaries seamlessly, an event mesh provides the necessary infrastructure. Often, an event mesh will require specific expertise and dedicated staff to maintain over time.

• For integrating a wide variety of applications and orchestrating complex business processes, an ESB offers comprehensive capabilities. Expect a high yearly cost for licensing a top-tier ESB.

• Consider a data storage solution to ensure data is not lost and can be replayed for various purposes. You will either need to specifically configure your message broker to push all messages into storage or select an option that provides durable message storage and querying capabilities.

Conclusion

Event-driven architecture offers a powerful paradigm for building scalable, responsive, and flexible applications. The choice of infrastructure plays a critical role in the success of an EDA implementation and may include message brokers, distributed streaming platforms, event gateways, event meshes, ESBs, and durable message storage. By understanding the strengths and use cases of each option, developers and businesses can make informed decisions that align with their specific needs, ensuring their architecture is robust, scalable, and capable of adapting to future challenges.