Kafka vs RabbitMQ vs SQS: Three Different Data Structures

Kafka, RabbitMQ and SQS are not three implementations of a queue. They are three different data structures, and choosing between them on throughput numbers is how teams end up building one on top of another.

The distinction that decides almost every real case is what happens to a message when it is read:

• Kafka — nothing. The message stays in the log. Consumers track a position, and can move it backwards.
• RabbitMQ — it is removed, once acknowledged. Consuming destroys.
• SQS — it is hidden, then deleted when you delete it. Also destructive.

If you need ten consumers to independently read the same events, or to reprocess last Tuesday after fixing a bug, only one of these does it natively. If you need to fan work out to whichever worker is free, the other two are simpler and cheaper.

Kafka: a partitioned, replayable log

Kafka is a distributed append-only log. Producers append; the log is retained for a configured period regardless of who has read it.

Consumers own their position. A consumer group commits offsets. Reset the offset and you reprocess from any point in the retention window. This is the property nothing else on this list gives you cheaply, and it is why Kafka dominates event sourcing, stream processing, and any pipeline feeding a data warehouse.

Partitions are the unit of parallelism and of ordering. Ordering is guaranteed within a partition, not across a topic. Messages with the same key land in the same partition and stay ordered relative to each other.

This is also Kafka's main constraint: you cannot have more active consumers in a group than partitions. Twelve partitions caps a consumer group at twelve consumers. Adding a thirteenth gets you an idle process. Partition count is a capacity-planning decision made up front, and while it can be increased later, doing so changes key-to-partition mapping and breaks ordering for existing keys.

What Kafka is bad at: per-message operations. There is no "delete this one message," no per-message TTL, no priority, no "retry this one in 30 seconds." A poison message at offset 4,001 blocks that partition until you deal with it, and dealing with it means a dead-letter topic and code you write yourself. If your workload is a task queue with retries and delays, Kafka will fight you the entire way.

Running it yourself on Kubernetes is a real project — see Kafka on Kubernetes with Strimzi.

RabbitMQ: a router with destructive reads

RabbitMQ is a message broker in the traditional sense. Producers publish to an exchange; the exchange routes to queues by binding rules; consumers take messages off queues and acknowledge them, at which point they are gone.

The routing is the point. Direct, topic, fanout and headers exchanges let the broker decide where a message goes based on its routing key. orders.eu.priority can land in three different queues by pattern match, configured at the broker, with no producer or consumer changes. Neither Kafka nor SQS has anything comparable — in Kafka, routing is a topic name plus consumer-side filtering.

Per-message control is excellent. Per-message TTL, priorities, delayed delivery via plugin, dead-letter exchanges, negative acknowledgement with requeue. This is a genuinely good task queue.

Quorum queues are the current replication model, built on the Raft consensus algorithm. Two things follow from that:

• Classic queue mirroring was removed in RabbitMQ 4.0. If you are running mirrored queues on 3.13.x, that is a migration you need to plan, not a config flag.
• Quorum queues are always durable and cannot be exclusive or non-durable. They trade latency for throughput, perform worse with large messages, and do not support global QoS prefetch — a channel cannot set one prefetch limit shared across all its consumers.

Quorum queues are explicitly not suited to temporary queues, workloads that create and delete queues at high frequency, or applications that do not use acknowledgements and publisher confirms. Those are real constraints; check your usage against them before migrating.

Streams are RabbitMQ's answer to the replay problem: an append-only, non-destructively-consumed log. Consumers attach at an offset with x-stream-offset, using first, last, a numeric offset, or a timestamp. They work over AMQP 0.9.1, though a dedicated binary protocol plugin is strongly recommended for full functionality and performance.

If you already run RabbitMQ and need replay for one use case, streams may save you from operating a second system. If replay is central to your architecture, Kafka's ecosystem — Connect, Streams, the entire data-platform integration surface — is far deeper.

SQS: a managed queue with real quotas

SQS is a fully managed queue. No brokers, no cluster, no capacity planning. You create a queue and use it.

That simplicity is the entire value proposition, and the trade is that you inherit AWS's semantics and quotas rather than choosing your own.

Standard queues are at-least-once with best-effort ordering. Nearly unlimited throughput, and duplicates will happen. Consumers must be idempotent — not as a best practice, as a correctness requirement.

FIFO queues give exactly-once processing and strict ordering within a message group. MessageGroupId is required, and it is both the ordering scope and the parallelism unit — messages in one group process sequentially, different groups process in parallel. The same trade-off as Kafka partitions, differently named.

The quotas that actually shape designs:

Two of these bite regularly. Retention maxes out at 14 days — SQS is not storage, and a consumer outage longer than your retention loses data permanently. And the 15-minute delay maximum means SQS alone cannot schedule work further out; that needs EventBridge Scheduler or Step Functions.

FIFO throughput is the quota people most often get wrong, because it is region-dependent:

• Without high-throughput mode: 300 transactions per second per API action, per partition. With batching, up to 3,000 messages per second (300 API calls × 10 messages).
• With high-throughput mode enabled, non-batched: 70,000 TPS in N. Virginia, Oregon and Ireland; 19,000 in Ohio and Frankfurt; 9,000 in Mumbai, Singapore, Sydney and Tokyo; 4,500 in London and São Paulo; 2,400 everywhere else.
• Batched, those become 700,000, 190,000, 90,000, 45,000 and 24,000 messages per second respectively.

The gap between 300 and 700,000 is entirely configuration and region. A FIFO design benchmarked in Ireland and deployed to a smaller region can hit a wall it never saw in testing.

For the wider AWS messaging picture, see SQS, SNS and EventBridge — the choice is frequently "which of these," not "queue or not."

Side by side

How to actually choose

Ask these in order. The first "yes" usually settles it.

1. Do consumers need to re-read history, or do multiple independent systems need every message? → Kafka. This is the replay requirement, and retrofitting it onto a destructive queue means building a log yourself.

2. Are you on AWS, and is this straightforward work distribution? → SQS. No cluster, no patching, no 3 a.m. page. Do not run a message broker to do what a managed queue does, and do not dismiss SQS as unsophisticated — "there is nothing to operate" is a large, durable advantage.

3. Do you need per-message behaviour — priorities, TTLs, delays, complex routing, sophisticated retry? → RabbitMQ. This is exactly what it is for, and both alternatives make you build it.

4. Is this high-volume streaming data feeding analytics or stream processing? → Kafka. The ecosystem is the reason, as much as the broker.

5. None of the above clearly applies? → SQS if you are on AWS, RabbitMQ otherwise. Both are simpler to operate than Kafka, and you can migrate later. Choosing Kafka "because we might need to scale" is the most common over-engineering mistake in this category — you take on a distributed stateful system to serve a requirement you do not have yet.

Mistakes that recur

Using Kafka as a task queue. No per-message ack, no priority, no delay, and head-of-line blocking within a partition. Teams end up writing a retry-topic ladder to simulate what RabbitMQ does natively.

Using RabbitMQ as an event store. Messages vanish when consumed. A new service that needs last month's events cannot have them. Streams help, but if this is the core requirement you picked the wrong tool.

Assuming SQS standard queues do not duplicate. They will. Idempotent consumers are not optional. This is the single most common SQS production bug, and it surfaces as mysterious double-charges and duplicate emails long after launch.

Treating partition count as tunable later. In Kafka it caps consumer parallelism, and increasing it breaks key ordering. In SQS FIFO the equivalent is MessageGroupId cardinality — too few groups and you have serialised your workload without noticing.

Ignoring the region in FIFO throughput planning. See the table above. A 25× difference between regions is not a rounding error.

Running Kafka because a benchmark said it was fastest. Throughput is rarely the binding constraint. Operational capacity usually is.

Frequently Asked Questions

Can Kafka replace RabbitMQ?

For event distribution, yes. For task queues with retries, priorities and delays, not without building those features yourself. Kafka has no per-message acknowledgement, so a failing message blocks its partition until you route it aside — typically to a dead-letter topic you implement and operate.

Can RabbitMQ replace Kafka?

Partly, through streams, which give append-only non-destructive consumption with offset-based re-reading. What you do not get is Kafka's ecosystem — Connect, Streams, and the integrations that make it the default backbone for data platforms. For one replayable use case alongside existing RabbitMQ, streams are a sensible way to avoid a second system.

Is SQS really unlimited?

Standard queues support a very high, near-unlimited rate of API calls per action. FIFO queues are explicitly limited, region-dependent, and vary by roughly 25× between the largest and smallest regions. If your design depends on FIFO throughput, check the limits for your specific region rather than the headline number.

What happened to RabbitMQ mirrored queues?

Classic queue mirroring was removed starting with RabbitMQ 4.0. Quorum queues, based on the Raft consensus algorithm, are the replacement. Migration tooling exists for 3.13.x, typically via a blue-green deployment, but check the constraints first — quorum queues are always durable, cannot be exclusive, and do not support global QoS prefetch.

How large a message can I send?

SQS accepts up to 1 MiB directly, and up to 2 GB using the Extended Client Library, which stores the payload in S3 and passes a reference. Kafka and RabbitMQ are configurable but both perform poorly with large messages. The pattern that works everywhere is the same: put the payload in object storage and send a pointer.

Do I need exactly-once delivery?

Almost certainly not — you need exactly-once effects, which is a different and much more achievable thing. An idempotent consumer gives you that with at-least-once delivery, and it keeps working through retries, redeliveries and replays. Designing for idempotency is cheaper and more robust than pursuing exactly-once delivery guarantees at the transport layer.

Which is cheapest?

For low to moderate volume, SQS — you pay per request with no idle cost. At sustained high throughput, self-managed Kafka or RabbitMQ usually wins on infrastructure cost, provided you do not count engineering time. Count engineering time. The crossover point is much further out than most teams estimate.

See also

• Kafka on Kubernetes with Strimzi — if you decided on Kafka and want to self-host
• SQS, SNS and EventBridge — choosing between the AWS messaging primitives
• KEDA Event-Driven Autoscaling — scaling consumers on queue depth
• AWS Step Functions — when the requirement is orchestration rather than messaging