Skip to the solution
MediumSystem Design11 min read

What is database replication?

537 views

The problem

Question presented to candidate: "What is database replication and why do we use it? Explain how a leader and its followers stay in sync, the difference between synchronous and asynchronous replication, and what can go wrong for users and for your data when replicas lag or the leader dies."

What a strong answer should cover:

  • Replication keeps copies of the same data on several machines for high availability, read scaling and locality โ€” and is not a backup (it copies mistakes too)
  • Single-leader: all writes go to the leader, which streams its change log (the WAL / binlog) to followers; reads may go to followers
  • Synchronous vs asynchronous commit: wait for a replica (slower writes, no loss of acknowledged writes if that replica is promoted) vs do not wait (fast, but a failover can lose recent acknowledged writes); quorum and semi-sync in between
  • Replication lag anomalies and fixes: read-your-writes (route to leader or track the write position), monotonic reads (stick a user to one replica)
  • Failover: detect, promote the most up-to-date follower, redirect clients, fence the old leader to avoid split brain
  • Multi-leader and leaderless (quorum) replication, and why they need conflict resolution

Clarifying questions expected:

  • Is losing the last second of acknowledged writes acceptable, or must every acknowledged write survive a failover?
  • Are users in one region or several, and what read and write volume is expected?

Code / implementation expected: Optional โ€” a clear explanation with examples; the example simulates a leader and two followers on a fake clock.

  • replication

Try it yourself

Sketch your approach before you read the solution. That is what the interview tests.

Stuck? Get a hint.

A short nudge toward the approach. It will not give the answer away.

Solution

The solution is hidden so you can try first

11 min read ยท 11 sections

Read the code

Leader-follower replication: commit latency, stale reads, read-your-writes and failover loss
// Single-leader replication on a fake clock (ms). The leader appends every write to its
// log (WAL) and streams it; a follower has applied an entry once `delay` ms have passed.
const FSYNC = 1;
const followers = { f1: 15, f2: 60 }; // one-way delay: f1 same region, f2 another region
const log = [];                        // entries: { lsn, t, key, value }

function write(t, key, value) { log.push({ lsn: log.length + 1, t, key, value }); return log.length; }
function appliedLsn(node, now) {
  if (node === "leader") return log.length ? log[log.length - 1].lsn : 0;
  let lsn = 0;
  for (const e of log) if (e.t + followers[node] <= now) lsn = e.lsn;
  return lsn;
}
function read(node, key, now) {
  const upTo = appliedLsn(node, now);
  let value = "(none)";
  for (const e of log) if (e.lsn <= upTo && e.key === key) value = e.value;
  return value;
}

console.log("1) Commit latency: when does the client hear OK?");
const rtt = (f) => 2 * followers[f];
console.log(`  async (leader fsync only):      ${FSYNC} ms`);
console.log(`  sync, wait for f1:              ${FSYNC + rtt("f1")} ms`);
console.log(`  sync, wait for f1 AND f2:       ${FSYNC + Math.max(rtt("f1"), rtt("f2"))} ms`);
console.log(`  quorum, any 1 of (f1, f2):      ${FSYNC + Math.min(rtt("f1"), rtt("f2"))} ms`);

console.log("\n2) Replication lag: Ana edits her bio at t=1000, then reloads at t=1030");
write(0, "bio:ana", "v1 hello");
const myLsn = write(1000, "bio:ana", "v2 new job");
for (const node of ["leader", "f1", "f2"]) {
  console.log(`  read from ${node.padEnd(6)} at t=1030: "${read(node, "bio:ana", 1030)}"  (applied lsn ${appliedLsn(node, 1030)})`);
}
function readYourWrites(key, now, tokenLsn) {
  for (const node of ["f2", "f1"]) if (appliedLsn(node, now) >= tokenLsn) return [node, read(node, key, now)];
  return ["leader", read("leader", key, now)];
}
for (const now of [1030, 1070]) {
  const [node, v] = readYourWrites("bio:ana", now, myLsn);
  console.log(`  read-your-writes (token lsn ${myLsn}) at t=${now}: served by ${node}, "${v}"`);
}

console.log("\n3) Monotonic reads: two reloads land on different replicas");
console.log(`  t=1040 via f1: "${read("f1", "bio:ana", 1040)}"`);
console.log(`  t=1045 via f2: "${read("f2", "bio:ana", 1045)}"   <- the page went back in time`);

console.log("\n4) Failover: writes every 5 ms from t=2000; the leader dies at t=2200");
for (let t = 2000; t < 2200; t += 5) write(t, `order:${t}`, "paid");
const CRASH = 2200;
for (const mode of ["async", "sync to f1"]) {
  // A write is acknowledged once the commit rule above is satisfied before the crash.
  const ackDelay = mode === "async" ? FSYNC : FSYNC + rtt("f1");
  const acked = log.filter((e) => e.t >= 2000 && e.t + ackDelay <= CRASH);
  for (const promoted of ["f1", "f2"]) {
    const has = appliedLsn(promoted, CRASH);
    const lost = acked.filter((e) => e.lsn > has).length;
    console.log(`  ${mode.padEnd(10)} promote ${promoted}: ${acked.length} writes acknowledged, ${lost} of them missing on the new leader`);
  }
}

Discussion

Other approaches, follow-ups and how it went in real interviews.

Sign in to join the discussion.

No comments yet. Share how you would answer it.

Question 33 of 106 in System design

Keep your momentum going.

Mark this question solved to track your progress, then take the next one while it is fresh.

Up nextMedium

Explain database indexing and B-trees

Continue