Event-Driven Architecture Fundamentals and Common Pitfalls (and How to Avoid Them)

Event-driven architecture (EDA) has been around for decades. However, it has recently been experiencing a resurgence as systems are becoming more complex and integrations span not only business units but between partners and third-party applications. Teams are now looking beyond synchronous integration techniques, such as web APIs, to EDA and asynchronous APIs such as Webhooks.

As teams transition from synchronous to asynchronous, it is important to be grounded in the fundamentals of messaging, EDA, and asynchronous APIs. In this article, we will explore these fundamental concepts and some design patterns to help you get started. We will also discuss pitfalls by teams and how to mitigate them by applying these principles and patterns.

A Review of Messaging Fundamentals

Let's review some messaging fundamentals to better understand and overcome some of the pitfalls encountered when applying event-driven architecture. While messaging fundamentals and EDA is a complex topic, we will simplify the concepts for the purposes of understanding. Refer to the Enterprise Integration Patterns website and associated book for a deep dive into messaging and event-driven architecture concepts commonly encountered in more complex enterprise and SaaS environments.

Messages and Events

Messages contain data that are published by a message producer to a message receiver (sometimes referred to as a message consumer or message processor). Receivers may be a local function or method, another process on the same host, or a process on a remote server. Communication may be direct between the message producer and receiver or managed by an intermediary, such as a message broker.

There are three common types of messages: commands, replies, and events.

• A command message requests work to be performed immediately or in the future. Command messages are often imperative: CreateAccount, SubmitPayment, and so on. Command messages are sometimes referred to as request messages.
• A reply message provides the result, or outcome, of a command message. Reply messages often add the suffix Result or Reply to differentiate them from their command counterparts: CreateAccountResult, SubmitPaymentReply, and so on. Reply messages are also referred to as response messages. Not all command messages result in a reply message.
• An event message is a specialized type of message that tells the message receiver about something that happened in the past. Events are used to indicate new data is available, data has been changed, a threshold has been exceeded, or the state has transitioned to a new value. A good event name uses past tense to indicate that an action has already taken place: AccountCreated, PaymentSubmitted, and so on.

EDA Pitfall #1: Note that messages are immutable, meaning that once they are published, they may not be modified.

Therefore, a message that requires modification or transformation to another format must be republished as a new message. Consider using correlation identifiers when this occurs to help with tracing and troubleshooting.

The Elements of a Message

Most of the focus of a message is placed on the message body. The message body typically uses a human-readable format, such as JSON or XML, though binary or plain text are also valid.

There is more to a message than just the message body, however. Messages may also include transport protocol semantics. Network protocols such as HTTP, MQTT, and AMQP include message headers with details such as creation timestamps, time-to-live (TTL), priority, and so on.

EDA Pitfall #2: When designing messages, keep in mind that the data format by itself does not fully describe a message. It must include all necessary information to process the message over the selected protocol. This is a common mistake made by those designing REST-based APIs without a foundation in the HTTP protocol. For example, developers end up reinventing cache controls and optimistic locking rather than leveraging what is already provided by the HTTP protocol.

Don't reinvent the wheel! Leverage the protocol format as part of your message design to avoid custom code that replicates what the protocol already supports.

Messaging Interaction Patterns

There are three common message interaction patterns found in EDA:

• Request-reply means that the message producer sends a message to a message receiver and waits while the receiver processes it and returns a reply. No work is performed by the producer while waiting for the response. This mimics a synchronous interaction approach.
• Fire-and-follow-up means that the message producer sends the message to the receiver, but the receiver may not process it immediately. The message producer is free to perform other tasks while waiting for a reply from the message receiver
• Fire-and-forget where the message producer sends the message but does not wait for a reply.

In addition, messages may be exchanged across different localities:

• Local messaging assumes that messages are sent and received within the same process. A "mailbox" sits between the code that produces the message and the code that will process the message. The consumer code processes each message as soon as possible, sometimes using threads or dedicated CPU cores to process multiple messages in parallel. Actor-based frameworks, such as Vlingo Xoom, support this kind of messaging.

• Interprocess messaging exchanges messages between separate processes but on the same host. Examples include UNIX sockets and dynamic data exchange (DDE).

• Distributed messaging involves two or more hosts for messaging. Messages are transmitted over a network using the desired protocol. Examples of distributed messaging include message brokers using Advanced Message Queuing Protocol (AMQP) for high-performance messaging, Message Queuing Telemetry Transport (MQTT) commonly found in edge computing and small footprint devices, along with API protocols and styles such as SOAP, REST-based APIs, Webhooks, and Websocket that leverage the uniquity of HTTP.

Finally, messages may have different delivery options depending on your needs. The most common options include:

• Direct/point-to-point routing, which enables a producer-to-receiver delivery. There is limited or no sophisticated routing performed in this case.
• Brokered routing leverages an intermediary, such as a message broker, to act as middleware for routing of messages to the correct destination. Most message brokers support a channel, which isolates message delivery to only those interested. A channel may contain routing rules to determine who should receive a message, filters that determine when a message should be routed to the receiver (and when to ignore it), and transformers that help adapt the message to different formats on behalf of the receiver.

EDA Pitfall #3: Don't ignore the decisions that need to be made when considering EDA. The combination of synchronous and asynchronous messaging styles, along with the locality of the messaging, determines the possibilities of a message-based solution. It will also guide your decisions on the kinds of middleware and routing support you will need to properly process and deliver your messages.

Just like EDA Pitfall #2, don't reinvent the wheel when embarking on the EDA journey. Consider how event gateways, message brokers, and other tools can help you focus on the solution rather than building out your own infrastructure by hand.

Common Event Design Patterns

When it comes to designing messages that will be exchanged by your system, there are many options. There exist many design patterns that can help you in your message design efforts while avoiding common pitfalls by teams starting their EDA journey. Let's look at a few common ones to help you get started successfully with EDA and asynchronous APIs.

Event Notification Design Pattern

Event notifications, sometimes referred to as ”thin events”, notify subscribers that a state change or business event has occurred. They seek to provide only the necessary information sufficient for the subscriber to determine if the event is of interest. It is then the responsibility of the event subscriber to fetch the latest data, often via an API.

{
  "eventType": "customerAddress.updated",
  "eventId": "123e4567",
  "updatedAt": "2020-01-14T03:56:45Z",
  "customerId": "330001003"
}

EDA Pitfall #4: Designing event payloads with all of the data at the time of the event is often considered a prudent thing to do. However, if the data changes between the time that the event message is published and the message is processed by the receiver, the processing will be based upon stale data. Instead, design the event message with limited data to prevent sharing stale data. Consider details such as the field that was changed, the old and new value, and any additional details that the message receiver will use to determine if further action needs to be taken or if the event should be ignored.

If you have ever paid for your electricity bill ahead of time but still receive an SMS notification that the bill is due, you have experienced this pitfall. The developers failed to check to see if the bill was paid before they sent the notification. Instead, they automatically sent the notification based on stale data and a pre-scheduled date. Applying this design pattern results in a better user experience if the message receiver doesn't just act on the event with the available stale data, but instead opts to fetch the latest details and avoids acting on the event entirely if it isn't necessary.

Event-Carried State Transfer Event Design Pattern

Event-carried state transfer events contain all available information at the time of the event. This is the opposite of the Event Notification pattern, which only provides minimal information with the event. Instead, the event message contains all available information at the time the event was published.

{
  "eventType": "customerAddress.updated",
  "eventId": "123e4567",
  "updatedAt": "2020-01-14T03:56:45Z",
  "customerId": "330001003",
  "previousBillingAddress": {
    "addressLine1": "...",
    "addressLine2": "...",
    "addressCity": "...",
    "addressState": "...",
    "addressRegionProvince": "...",
    "addressPostalCode": "..."
  },
  "newBillingAddress": {
    "addressLine1": "...",
    "addressLine2": "...",
    "addressCity": "...",
    "addressState": "...",
    "addressRegionProvince": "...",
    "addressPostalCode": "..."
  },
  ...
}

EDA Pitfall #5: Event-carried state transfer pattern may address the scenario where other teams that will be message receivers find your event message design lacking important fields for successfully processing the message. There are a few reasons why this pattern may address this challenge:

• Subscribers wish to process the event using a snapshot of the data at the time of the event's publication, rather than at the time of processing the event. This is quite different from the event notification pattern that encourages retrieving the current data at the time of processing. A common scenario is synchronizing state between systems.
• Supporting the replay of data state changes over time, rather than simply reacting to any change that has occurred. Event sourcing is an example of using this pattern.
• Reduces API traffic load when event processing does not require the most recent data snapshot, as message receivers are able to perform processing with the data provided within the event.

Using Hypermedia Links in Events

Event messages often correspond to at least one existing web API, perhaps more than one. There may be an API operation that triggers the event message to be published, and/or an API operation that may be used to retrieve the latest data associated with the event. Embedding hypermedia links into the event message body allows the event receiver to easily identify and use the associated web API as part of the processing.

{
  "eventType": "customerAddress.updated",
  "eventId": "123e4567",
  "updatedAt": "2020-01-14T03:56:45Z",
  "customerId": "330001003",
  "_links": [
    { "rel": "self", "href":"/events/123e4567" },
    { "rel": "customer", "href":"/customers/330001003" }
  ]
}

EDA Pitfall #6: Lack of connections between your event messages and your web APIs can lead to confusion or a misunderstanding about why the event was generated or how to properly retrieve any associated data related to the event.

Add hypermedia links into your event message body enables event processing to easily identify the API associated with the received event.

Event Batching

While most event messages are sent one-at-a-time, some designs may benefit from grouping events into a batch. This is especially the case when message receivers are catching up on events after being offline for some period of time. A simple approach to supporting event batching is to wrap the notification with an array and enclose each message within the response, even if there is only one event message at the time.

[
  {
    "eventType": "customerAddress.updated",
    "eventId": "123e4567",
    "updatedAt": "2020-01-14T03:56:45Z",
    "customerId": "330001003",
    "_links": [
      { "rel": "self", "href":"/events/123e4567" },
      { "rel": "customer", "href":"/customers/330001003" }
    ]
  },
  ...,
  ...
]

EDA Pitfall #7: The developer experience of processing a large number of messages is often overlooked when a large number of event messages published to a Webhook endpoint might overwhelm the HTTP receiver with hundreds or thousands of HTTP POST requests to process. Of course, event batching requires that subscribers handle one or more messages within each notification. While rate limiting can also address this issue, it will result in delays to delivery that batching can overcome.

Wrap-up

In this article, we looked at seven pitfalls encountered when beginning your journey to an event-driven architecture. Some of these pitfalls are rooted in the fundamentals of EDA, while others are focused more on the design of event messages and the resulting developer experience.