Skip to the solution
HardSystem Design18 min read

How would you design S3-like object storage?

0 views

The problem

Question presented to candidate: "Design an object storage service like Amazon S3. Customers create buckets and PUT, GET, LIST and DELETE objects from a few bytes to many terabytes. It has to be extremely durable, survive losing a whole data centre, and scale to hundreds of petabytes. Walk me through the API, how metadata and data are stored, how you keep data safe, and what breaks first as it grows."

What a strong answer should cover:

  • Split into a metadata plane (bucket and key to object record: size, checksum, version, shard locations) and a data plane (storage nodes holding immutable chunks or shards), with stateless front ends in between
  • Durability with erasure coding instead of plain replication: RS(6,3) stores 150% of the data and survives 3 lost shards, versus 300% for 3 copies; repair reads k shards, so repair traffic is the cost
  • Placement across failure domains: shards spread so that losing one availability zone loses at most m shards (RS(6,3) as 3/3/3 survives; RS(10,4) over 3 zones does not)
  • The write path and consistency: write and acknowledge shards first, then commit the metadata record as the single commit point; that gives strong read-after-write and atomic overwrites, plus conditional writes
  • Metadata at scale: range-partitioned by bucket and key so LIST by prefix is a range scan, split hot partitions, versioning and delete markers, garbage collection of unreferenced data
  • Capacity numbers: objects, bytes, disks, metadata size, throughput, and what breaks first at 10x (metadata hot spots, repair bandwidth, small-object overhead)

Clarifying questions expected:

  • Object size distribution, request rates, and read/write ratio?
  • Durability and availability targets, and must it survive a full AZ or region loss?
  • Which features are in scope: versioning, listing, multipart, lifecycle, storage classes?

Code / implementation expected: Optional โ€” architecture and numbers come first; the example implements Reed-Solomon over a small prime field to show encode, loss and rebuild, and computes the overhead and loss odds.

  • system-design
  • case-study

Try it yourself

Sketch your approach before you read the solution. That is what the interview tests.

Stuck? Get a hint.

A short nudge toward the approach. It will not give the answer away.

Solution

The solution is hidden so you can try first

18 min read ยท 18 sections

Read the code

Reed-Solomon RS(6,3) erasure coding: encode, lose shards, rebuild, then overhead, loss odds and AZ placement per scheme
// Erasure coding for an object store: systematic Reed-Solomon (k=6, m=3) over the prime field GF(257),
// then overhead, loss probability and AZ placement for several schemes.
const P = 257;
const mod = (a) => ((a % P) + P) % P;
function inv(a) { let r = 1, b = mod(a), e = P - 2; while (e > 0) { if (e & 1) r = (r * b) % P; b = (b * b) % P; e >>= 1; } return r; }
// Evaluate at x the polynomial through points (xs[i], ys[i]) (Lagrange interpolation).
function interp(xs, ys, x) {
  let s = 0;
  for (let j = 0; j < xs.length; j++) {
    let num = 1, den = 1;
    for (let i = 0; i < xs.length; i++) if (i !== j) { num = mod(num * (x - xs[i])); den = mod(den * (xs[j] - xs[i])); }
    s = mod(s + ys[j] * num % P * inv(den));
  }
  return s;
}
const K = 6, M = 3, N = K + M;
// Shard i (1..9) holds the value at x=i for every stripe; shards 1..6 are the data itself.
function encode(bytes) {
  const L = Math.ceil(bytes.length / K), shards = [];
  for (let i = 0; i < N; i++) shards.push([]);
  for (let s = 0; s < L; s++) {
    const xs = [], ys = [];
    for (let i = 0; i < K; i++) { xs.push(i + 1); ys.push(s * K + i < bytes.length ? bytes[s * K + i] : 0); }
    for (let i = 0; i < N; i++) shards[i].push(i < K ? ys[i] : interp(xs, ys, i + 1));
  }
  return shards;
}
function decode(shards, alive, len) {
  if (alive.length < K) return null;
  const use = alive.slice(0, K), out = [];
  for (let s = 0; s < shards[use[0] - 1].length; s++) {
    const ys = use.map((x) => shards[x - 1][s]);
    for (let i = 0; i < K; i++) out.push(interp(use, ys, i + 1));
  }
  return out.slice(0, len);
}
const text = "object stores keep bytes safe by math";
const bytes = [...text].map((c) => c.charCodeAt(0));
const shards = encode(bytes);
console.log(`== 1. RS(6,3): ${bytes.length} bytes -> 9 shards of ${shards[0].length} symbols ==`);
for (let i = 0; i < N; i++) console.log(`shard ${i + 1} ${i < K ? "data  " : "parity"} [${shards[i].join(", ")}]`);
for (const lost of [[2, 5, 8], [1, 2, 3], [1, 4, 7, 9]]) {
  const alive = [];
  for (let i = 1; i <= N; i++) if (!lost.includes(i)) alive.push(i);
  const got = decode(shards, alive, bytes.length);
  const msg = got ? `rebuilt from [${alive.slice(0, K).join(",")}] -> "${String.fromCharCode(...got)}"` : `cannot rebuild: ${alive.length} shards left, need ${K}`;
  console.log(`lose [${lost.join(",")}]: ${msg}`);
}

// 2. Compare schemes. Assumption: each shard is lost before repair finishes with probability p = 0.001, independently.
function sci(x) {
  let e = Math.floor(Math.log10(x)), m = x / Math.pow(10, e);
  if (Number(m.toFixed(2)) >= 10) { m /= 10; e += 1; }
  return `${m.toFixed(2)}e${e}`;
}
function choose(n, r) { let c = 1; for (let i = 1; i <= r; i++) c = (c * (n - r + i)) / i; return c; }
function lossProb(n, k, p) { let s = 0; for (let i = n - k + 1; i <= n; i++) s += choose(n, i) * Math.pow(p, i) * Math.pow(1 - p, n - i); return s; }
console.log("== 2. Overhead and loss probability (p = 0.001 per shard per repair window) ==");
const schemes = [["3x replication", 3, 1], ["RS(4,2)", 6, 4], ["RS(6,3)", 9, 6], ["RS(10,4)", 14, 10]];
for (const [name, n, k] of schemes) {
  const raw = (100 * n / k).toFixed(0);
  console.log(`${name.padEnd(14)} stores ${String(raw).padStart(3)}% of the data, survives ${n - k} lost, P(loss) ${sci(lossProb(n, k, 0.001))}, repair cost ${k}x`);
}

// 3. Spread n shards round-robin over 3 availability zones: does the object survive losing the fullest AZ?
console.log("== 3. Placement over 3 AZs, lose one whole AZ ==");
for (const [name, n, k] of schemes) {
  const per = [0, 0, 0];
  for (let i = 0; i < n; i++) per[i % 3]++;
  const worst = Math.max(...per);
  console.log(`${name.padEnd(14)} AZs hold ${per.join("/")} -> lose ${worst}, ${worst <= n - k ? "still readable" : "DATA UNAVAILABLE"}`);
}

Discussion

Other approaches, follow-ups and how it went in real interviews.

Sign in to join the discussion.

No comments yet. Share how you would answer it.

Question 99 of 106 in System design

Keep your momentum going.

Mark this question solved to track your progress, then take the next one while it is fresh.

Up nextHard

How would you design a metrics monitoring and alerting system?

Continue