MediumSystem Design17 min read

Kafka vs SQS — how do you choose for an event-driven system in 2025?

Q: Kafka vs SQS — how do you choose for an event-driven system in 2025?

**Target Audience:** Beginners to messaging — you know what a server and a database are; queues, logs, offsets and consumer groups are defined as we go. **Difficulty:** Medium **Verified with:** Node 22.22 and Python 3.12 (both print identical output). Product limits are from the AWS SQS Developer Guide and the Apache Kafka release announcements, checked in October 2026. > **How to read this doc:** Every idea is explained in plain language first, then tagged with `📌 Interview term:` so you also learn the exact words an interviewer expects. The run in section 5 comes from the example program; the **Verified** section quotes its full output. ## 1. Why This Even Matters — A Story First Think of two ways a busy restaurant can pass orders from the waiters to the kitchen. **The ticket rail.** A waiter clips a paper ticket to a rail. A cook takes it down, cooks it, and throws the ticket in the bin. If the cook drops the pan, the ticket goes back on the rail and someone else picks it up.

370 views

40 of 106

The problem

Question presented to candidate: "We are building an event-driven backend: orders, payments, notifications, analytics. Some people want Kafka, some want SQS. How do you decide between them, and would you ever use both?"

What a strong answer should cover:

The core difference in model: Kafka is a partitioned, replicated log that consumers read by offset and can re-read; SQS is a managed queue where each message is handed to one consumer, hidden by a visibility timeout, and deleted once processed
What follows from that: Kafka gives replay, many independent consumer groups over the same data, and per-partition ordering, with parallelism capped by the partition count; SQS gives per-message retries, a dead-letter queue, elastic consumers and almost no operations, with ordering only in FIFO queues per message group
Delivery: both are at-least-once in normal use, so consumers must be idempotent; SQS standard can also deliver out of order
Current limits checked against the docs: SQS keeps messages at most 14 days and accepts payloads up to 1 MiB (raised from 256 KiB in August 2025); FIFO throughput depends on high-throughput mode and region; Kafka 4.0 (March 2025) runs without ZooKeeper, and Kafka 4.2 made share groups (queue-style consumption, KIP-932) production-ready
A decision based on requirements (replay, fan-out, ordering, throughput, team and cloud), and the common hybrid: Kafka (or a managed Kafka) as the event backbone, SQS for task queues around it

Clarifying questions expected:

Do several teams need to consume the same events independently, and do we ever need to replay history?
What ordering is required, and at what granularity (per order, per user, global)?
What is the peak throughput, and are we all-in on AWS with a small platform team?

Code / implementation expected: Optional — a small simulation of both consumption models (offsets and consumer groups vs visibility timeout, redelivery and DLQ); the example does exactly that.

kafka
sqs
event-driven

Try it yourself

Sketch your approach before you read the solution. That is what the interview tests.

Stuck? Get a hint.

A short nudge toward the approach. It will not give the answer away.

Solution

Target Audience: Beginners to messaging — you know what a server and a database are; queues, logs, offsets and consumer groups are defined as we go. Difficulty: Medium Verified with: Node 22.22 and Python 3.12 (both print identical output). Product limits are from the AWS SQS Developer Guide and the Apache Kafka release announcements, checked in October 2026.

How to read this doc: Every idea is explained in plain language first, then tagged with 📌 Interview term: so you also learn the exact words an interviewer expects. The run in section 5 comes from the example program; the Verified section quotes its full output.

1. Why This Even Matters — A Story First

Think of two ways a busy restaurant can pass orders from the waiters to the kitchen.

The ticket rail. A waiter clips a paper ticket to a rail. A cook takes it down, cooks it, and throws the ticket in

The solution is hidden so you can try first

17 min read · 12 sections

Read the code

Log vs queue: Kafka-style partitions, offsets and consumer groups vs SQS-style visibility timeout, redelivery and DLQ

Open in playground

// Log (Kafka-style) vs queue (SQS-style), simulated in-process with a fake clock.
const PARTS = 3;
const part = (key) => { let h = 0; for (const c of key) h = (h * 31 + c.charCodeAt(0)) % 1000003; return h % PARTS; };

console.log("== A. Kafka-style log: partitions, offsets, consumer groups ==");
const log = [[], [], []];
for (const o of ["o1", "o2", "o3", "o4"])
  for (const step of ["created", "paid", "shipped"]) log[part(o)].push(o + ":" + step);
for (let p = 0; p < PARTS; p++) console.log("partition " + p + ": " + log[p].join(" "));

function assign(consumers) { // round-robin partitions over the group members
  const a = {}; consumers.forEach((c) => (a[c] = []));
  for (let p = 0; p < PARTS; p++) a[consumers[p % consumers.length]].push(p);
  return a;
}
function runGroup(name, consumers, offsets) {
  const a = assign(consumers); let read = 0;
  for (const c of consumers) {
    const got = [];
    for (const p of a[c]) { while (offsets[p] < log[p].length) { got.push(log[p][offsets[p]]); offsets[p]++; read++; } }
    console.log("  " + name + "/" + c + " partitions [" + a[c].join(",") + "] read " + got.length + (got.length ? ": " + got.join(" ") : " (idle)"));
  }
  return read;
}
const billing = [0, 0, 0], analytics = [0, 0, 0];
console.log("group billing (2 consumers):");
const b1 = runGroup("billing", ["c1", "c2"], billing);
console.log("group analytics (1 consumer) reads the same log independently:");
const a1 = runGroup("analytics", ["a1"], analytics);
console.log("billing read " + b1 + ", analytics read " + a1 + ", log still holds " + log.flat().length + " events");
billing[0] = 0; billing[1] = 0; billing[2] = 0;
console.log("bug fixed: billing resets offsets to 0 and replays:");
console.log("  replayed " + runGroup("billing", ["c1", "c2"], billing) + " events");
console.log("group of 4 consumers on 3 partitions:");
runGroup("wide", ["w1", "w2", "w3", "w4"], [0, 0, 0]);

console.log("");
console.log("== B. SQS-style queue: visibility timeout, redelivery, DLQ ==");
const VIS = 30, MAX_RECEIVES = 3, WORK = 10;
const queue = [];
for (let i = 1; i <= 8; i++) queue.push({ id: "j" + i, visibleAt: 0, receives: 0, deleted: false });
const dlq = [], doneLog = [], counts = {};
const workers = [{ name: "w1", busyUntil: 0, job: null }, { name: "w2", busyUntil: 0, job: null }, { name: "w3", busyUntil: 0, job: null }];
for (let t = 0; t <= 200; t += 5) {
  for (const w of workers) {
    if (w.job && t >= w.busyUntil) {
      const m = w.job; w.job = null;
      if (m.id === "j6") console.log("t=" + t + " " + w.name + " " + m.id + " FAILED (poison message), not deleted");
      else if (m.deleted) console.log("t=" + t + " " + w.name + " " + m.id + " finished, but it was already deleted: duplicate work");
      else {
        m.deleted = true; counts[m.id] = (counts[m.id] || 0) + 1; doneLog.push(m.id);
        console.log("t=" + t + " " + w.name + " " + m.id + " done, DeleteMessage");
      }
    }
  }
  for (const w of workers) {
    if (w.job) continue;
    const m = queue.find((x) => !x.deleted && !x.inDlq && x.visibleAt <= t);
    if (!m) continue;
    if (m.receives >= MAX_RECEIVES) { m.inDlq = true; dlq.push(m.id); console.log("t=" + t + " " + m.id + " received " + m.receives + " times, moved to DLQ"); continue; }
    m.receives++; m.visibleAt = t + VIS; w.job = m;
    const slow = m.id === "j2" && m.receives === 1; // the first copy of j2 stalls (a 45 s GC pause)
    w.busyUntil = t + (slow ? 45 : WORK);
    if (m.receives > 1) console.log("t=" + t + " " + w.name + " receives " + m.id + " again (receive #" + m.receives + ")");
  }
}
console.log("completion order: " + doneLog.join(" "));
const completed = Object.keys(counts).length;
console.log("jobs completed: " + completed + " of 8, in DLQ: [" + dlq.join(",") + "], queue empty: " + queue.every((m) => m.deleted || m.inDlq) + ", replay possible: false");

Discussion

Other approaches, follow-ups and how it went in real interviews.

No comments yet. Share how you would answer it.

Question 40 of 106 in System design

Keep your momentum going.

Mark this question solved to track your progress, then take the next one while it is fresh.

All System design questions

Up nextMedium→

What is the circuit breaker pattern?

Continue