Introduction to Messages Queues

Modern software and system architectures encourage the decoupling of different layers/services in an application's design. These services and/or layers can be web service APIs, database systems, cache proxy servers, serverless functions, etc. Communication between these elements is done by sending data in the form of messages from one point to another, oftentimes asynchronously. In this post, we will look at one of the tools that helps broker messages between the layers and services in these architectures and ensure that messages are transferred at a rate that can be handled by the intended destination service/layer.

What are message queues?

A message queue is a component in a system's architecture that helps buffer and distribute asynchronous requests. As stated earlier, a message is sent in a request from one point to another; for example, from a front-end application to a back-end API. A message is a serialized data construct, mostly in XML or JSON format, and it contains all the information required to perform the requested operation.

1. Message queues maintain an internal queue of messages that are distributed on schedule to the intended destination.
2. Messages that cannot be consumed fast enough line up in the queue until the destination service is ready to process more messages.
3. Message queues sit in between two services/layers that need to communicate with one another. The component that makes the request by adding a message to the queue is known as a message producer, while the component that picks up the messages from the queue and does the main processing is known as the message consumer.
4. A message queue arranges messages in a sequence to be delivered to the consumers who can then consume messages from the queue.

Let's take a look at message producers and consumers in more detail in the next section.

Producers and consumers

Message producers create messages which are buffered by the message queue and delivered to the message consumers, who then perform the asynchronous processing on behalf of the producer. Think of this like making an order at a restaurant. You and your friends (message producers) place your orders (message requests) and deliver them to the waiter (message queue) who holds and delivers all the orders to the kitchen (message consumer), where the actual preparation of the meals you ordered takes place.

In an architecture that is designed to scale easily, producers and consumers run as separate processes and are often hosted on separate machines.

Producers and consumers function as independent systems and the only contract that exists between them is the message format with which they are to communicate. The producer must send messages in the format (JSON or XML) and structure expected by the consumer as message queues do not perform message transformation.

Producers and consumers are decoupled and exist separately, but the producer has a dependency on the consumer. Most consumers are completely independent components that often only specialize in performing a single task, for example, an email service that receives email requests and its sole responsibility is to send emails to different addresses.

In a well-decoupled system, a consumer should know nothing about the producer and only depend on valid messages that the queue delivers to it.

Advantages of using message queues

Now that we understand the responsibility of message queues and how they work with message producers and consumers, let's take a look at some of the advantages of using message queues.

1. Because producers and consumers are independent components, they can be implemented in different languages.
2. By hosting producers and consumers on separate machines and decoupling them, it is easy to scale each side of the message queue separately. You can add more consumers to speed up the rate of processing.
3. They bring about asynchronous processing which ensures that, unlike the typical request/response synchronous pattern, producers can carry on with other tasks after making the request and do not have to wait for a response to be received before doing any other work.
4. They can be used to implement asynchronous processing in a system that was built to be synchronous.
5. Job submission, which is the process by which producers submit messages to the queue, is very fast compared to sending messages directly to the consumer, and therefore producers can quickly make requests and carry on with other tasks.
6. Because of the scheduled delivery of messages to consumers, message queues help even out traffic spikes, making sure that consumers are not overloaded when there is a sudden increase in requests.
7. With the decoupled nature of producers and consumers, failure at one end does not affect the other. If producers are down, consumers can keep consuming and processing messages from the queue. If consumers are down, producers can keep adding requests to the queue until consumers are back up and running.

Implementing messages queues

So how do we implement a message queue and take advantage of its benefits? One thing to note is that decoupling of systems with message queues increases the complexity of the system's architecture, so that's one trade-off you will need to consider.

One major advantage to the implementation task is that there are open source frameworks and services out there that can help reduce the burden of setting up message queues.

Frameworks are used when complete control of the queue setup is required and the expertise is available to get it up and running. However, if you want some hand-holding and want to set up message queues quickly, it's advisable to go for the third-party services which might come at an extra cost.

Some of these frameworks and third-party services are listed below.

Frameworks

• RabbitMQ
• Apache Kafka
• ZeroMQ

Services

• Hookdeck
• Google Cloud Pub/Sub
• IBM MQ
• Amazon MQ

Conclusion

In this post, we have taken a high-level look at message queues without worrying so much about the implementation details. Message queues are a very important part of modern software and systems architectures, as they help introduce asynchronous processing between different layers and components in the architecture, and today can be found in almost every scalable architecture.

To learn more about the internal workins of messages queues, checkout our article Message Queues: Deep Dive.

‍