Message Queues vs. Logs

One-Liner

The two main paradigms for asynchronous messaging: traditional message queues where messages are deleted after being consumed, and logs where messages are immutable and can be replayed.

What It Is

Message Queue: A queue of messages where multiple producers can add messages and multiple consumers can process them. Once a message is successfully processed, it is removed from the queue. (e.g., RabbitMQ, SQS).
Log: An append-only, immutable sequence of records. Consumers read from the log at their own pace. Messages are not deleted after being read, but are retained for a configurable period. (e.g., Apache Kafka).

Why It Exists

To provide different models for asynchronous communication. Queues are great for traditional work distribution, while logs are powerful for event streaming and building systems that can replay history.

How It Works

Queue: Consumers “compete” for messages. Each message is typically delivered to only one consumer.
Log: Consumers (in a consumer group) read from a specific offset in the log. Multiple consumer groups can read from the same log independently.

Tradeoffs

Queue

Pros: Simple model for work distribution, good for transient messages.
Cons: Messages are ephemeral, hard to replay events.

Log

Pros: Durable, allows for replaying messages, multiple consumers can read the same data for different purposes.
Cons: More complex to manage, requires consumers to manage their own offset.

Failure Modes

Queue: A “poison pill” message can get stuck in a retry loop and block other messages.
Log: A slow consumer can fall far behind the head of the log, leading to a large “consumer lag”.

Interview Traps

Not understanding the fundamental difference in how messages are consumed.
Not being able to explain when to use one over the other.

Real-World Usage

Queue: Distributing tasks to a pool of workers.
Log: Building a real-time activity feed, change data capture from a database.

Anti-Patterns

Using a log when you just need simple, transient task distribution.
Using a queue when you need to be able to replay messages or have multiple independent consumers of the same data.

Publish/Subscribe (Pub/Sub) Pattern
Event Sourcing
Change Data Capture (CDC)