Message queues and event streams are key components of event-driven architecture, but what exactly are the similarities and differences between them?Which one of them should be chosen under what circumstances?
Translated from Choosing Between Message Queues and Event Streams, Tun Shwe is Vice President of Data at Quix, where he leads data strategy and developer relations. He focuses on helping companies envision and execute strategic data visions with the cutting edge of stream processing. before him. Implementing event-driven architecture (EDA) is a challenging task. One of them is choosing the right tool for the job. Many event-driven tools may seem similar at first glance, and you might expect them to work equally well for the same purpose. But that's often not the case, and choosing the best solution for your needs can be tricky.
Take, for example, messaging technology. Choosing a message may seem simple at first. However, "messaging" is an umbrella term often used to describe different types of components, such as event buses, publish-subscribe messaging services, message queue systems, and event streaming platforms.
While there is some overlap in the capabilities and use cases of all of these components, there are also a number of significant differences. Before embracing one of these messaging technologies (or a mix of them), it's crucial to understand these differences.
I'll focus on message queues and event streams, highlighting their differences, commonalities, and applicability to various use cases. As a follow-up reading, I recommend reading the "Guide to the Event-Driven, Event Streaming Stack", which covers all the components of EDA and walks you through a reference use case and decision tree to understand the scenarios for which each component is applicable. Before we discuss message queues and event streams, let's first clarify what "messages" and "events" mean. Message is a general term used to describe packets sent from one component to another. There are different types of messages, including:
Command messages。It carries instructions from the receiver to perform a specific action.
Query messages。A request for information from a component.
Reply to messages。The server or receiver replies to the request query message.
Transactional messages。Used in systems where messages are part of a transaction and must be processed in a reliable, often atomic, manner.
Here's a basic example that demonstrates a command message that instructs the banking system to initiate a funds transfer:
At the same time, an event is a significant event or state change within the system. A button in the UI is clicked, a motion sensor records movement, or a payment is successfully processed – these are all examples of events. When an event "propagandizes" between the components of a system, it does so in the form of a message, so an event is a type of message.Below is an example of an event message that documents that the above command message has been processed and the funds have been successfully transferred between accounts.
Because events are packaged into messages, you'll often hear the terms "message" and "event" used interchangeably when discussing event-driven architecture. However, it's worth noting that while events are messages, not all messages are events.Now, let's move on to message queues and event streams. Message queues operate by providing temporary storage for messages that are about to be processed by consumers. The producer sends the message to the message, which stores it in a queue. Consumers retrieve messages from a queue, typically in first-in, first-out (FIFO) order. Once consumed from the queue (and acknowledged), the message is deleted. This setup decouples the components, ensuring that messages are reliably and orderly processed by consumers.
Overview of message queues.
Similar to message queues, event streams revolve around producers, consumers, messages, and messages. However, there are some significant differences compared to message queues:
Event streaming involves a continuous flow of event messages. (Typically, using a message queue doesn't involve such a high amount of data and rate).
*Event messages are typically stored in a topic (or channel). Unlike peer-to-peer queues, where a single recipient consumes each message, topics use a publish-subscribe model that allows multiple consumers to read the same message.
Messages can be stored sequentially for a longer period of time. (They are not discarded immediately after being consumed).
The main purpose of a message queue is to reliably deliver messages from point A to point B, while event streams follow a different paradigm. Event streams do do do the same, but in addition to distribution, it's often transformed in real-time before the event data is delivered to the target (so the high-level process is a > data transformation > b). Data transformation typically involves the use of stream processing technologies such as Kafka Streams or Apache Flink.
Overview of event streams.
There are a number of differences between the technologies that allow you to implement event streams and those used for message queues. To highlight these differences, I'm going to compare Apache Kafka (an event streaming platform) and RabbitMQ (which provides messages for message queues**) I chose Kafka and RabbitMQ especially because they are popular, widely used solutions that offer rich functionality and have been extensively battle-tested in production. They are considered by many to be the standard. 
The above table is just a brief comparison of Kafka's event stream processing capabilities and RabbitMQ's message queues, and summarizes the basic differences between them. However, if you'd like to take a deeper look at how the two technologies compare (including other criteria such as architecture, developer experience, and ecosystem), check out this Kafka vs. RabbitMQ blog post. Both message queues and event streams can be used in scenarios that require decoupling, asynchronous communication between different parts of the system. For example, in a microservices architecture, both can provide low-latency messaging between components. However, beyond messaging, event streaming and message queues have their own advantages and are suitable for different use cases.
Message queuing technology is typically used for:
Communication between components written in different languages and between "using" different protocols. Message queuing solutions like RabbitMQ and ActiveMQ achieve this by supporting multiple protocols and programming languages.
Use cases that require complex message routing (e.g., the trading platform routes buy and sell orders to different processing queues depending on the type and size of the order).
Distribute tasks across minions, where each task is processed only once by a single consumer.
Deal with consumers who disconnect frequently. In these cases, a message queue system is a good choice because of its ability to sort messages, temporary persistence, and resend messages.
Thousands of companies around the world have incorporated message queuing technology into their technology stacks. For example, you can check out the RabbitMQ Summit** to see how organizations of all shapes and sizes are using RabbitMQ message queues in production. **There are many discussions on it for you to browse (just click on the "Past Events" drop-down menu and select the summit version of your choice to see all related discussions). Now, let's move on to event stream processing, which is a good fit for:
Collect, persist, and transmit large streams of events, such as clickstream data, marketplaces, and high-frequency readings from IoT devices and sensors.
Continuously process and analyze data to provide actionable insights and support real-time decision-making (e.g., analyzing financial transactions as they occur to identify and mitigate fraud as quickly as possible).
Event sourcing. In this case, a technology like Kafka is ideal because its immutable and append-only log structure ensures a reliable, ordered, and replayable record of events. This allows the complete sequence of historical state changes to be stored and queried.
Log aggregation use cases. Event stream processing solutions are a suitable choice because they typically offer good performance, strong durability guarantees, and low latency. In addition, event stream processing technologies are often integrated with (or provide a direct way to integrate) with many other systems, making it easy to ingest log data from different components.
There are countless companies that are taking advantage of event stream processing. For example, some large organizations, including Uber, PayPal, and Netflix, have shared how and why they use Kafka and the benefits they receive. It's worth reading about their experiences. But it's not just large enterprises that rely on event stream processing. Check out my last blog post to learn how small and medium-sized companies can take advantage of Kafka's event stream processing capabilities. Note that some event-driven architectures use both event stream processing and message queues. For example, in the case of an e-commerce platform, you can use event stream processing to collect and analyze clickstream data from users in real-time in order to provide them with relevant banners, offer discounts, and product recommendations. At the same time, message queuing solutions can be used to queue orders for payment and processing.
For many messaging use cases, message queues are a good choice. If you're early in your event-driven journey, message queuing technology is often easier to deploy and manage than event stream processing solutions, making it compelling. However, despite the added added complexity, organizations sometimes migrate from message queues to event streaming. The main reason?Scalability, reliability, and performance.
That's exactly what happened to Doordash, AppDirect, and global payment providers. They all initially used message queuing technology, specifically RabbitMQ and ActiveMQ. However, in the face of growing data volumes, these message queue systems experienced serious scalability, reliability, and performance issues. All three companies eventually had to re-evaluate their technology stacks and replace RabbitMQ ActiveMQ with Kafka's event stream processing capabilities. By moving to Kafka, the three organizations have significantly improved system uptime, scalability, availability, and performance (lower latency and higher throughput). I'm curious to see if more businesses will continue to move from message queues to event streaming in the future. Another possible trend is that companies will adopt an event streaming platform from the start, especially given the introduction of Kafka's queue scheduling. In other words, Kafka could eventually become a technology that works equally well for event streaming use cases and traditional message queuing scenarios (currently challenging to use Kafka as a traditional message queue – see the details in this article for details). If you're dealing with small and medium-sized workloads and want to reliably and flexibly route messages between components, and your system is primarily concerned with the current state, message queuing technology is a good choice.
On the other hand, if you want to handle high-volume, high-frequency event streams in a scalable and reliable way, and you need complex processing to gain real-time insights when data arrives, and your system is not only focused on the current state, but also the history of state changes, then event stream processing is the right choice.
As we've seen, sometimes companies start using only message queues and then migrate to event streaming technology. This migration is very difficult and time-consuming. So, if you're in the early stages of your event-driven journey and you're thinking about whether event streaming or message queues are the right choice for you, ask yourself: can the current needs be met equally by both?If the answer is yes, then I recommend that you go for Event Streaming. It is a stronger, more reliable foundation for the future.
It's true that event streaming tools are often more difficult to learn and manage than message queues. But don't be discouraged by that. Hosting platforms such as Confluent Cloud and Redpanda greatly simplify the work of handling event streams. In addition, they work seamlessly with serverless stream processing solutions like Quix, enabling you to easily build, deploy, and monitor event streaming applications that extract value from real-time data. Check out these interactive templates to see what event-driven applications you can create by combining Confluent Cloud Kafka Redpanda as streaming and Quix as the streaming engine.