Skip to the solution
MediumSystem Design14 min read

Why prefer stateless services and how do you handle sessions?

566 views

The problem

Question presented to candidate: "We want to run our web app on many instances behind a load balancer and autoscale it. Why does everyone say the instances should be stateless? Users still log in and have carts, so where does that session data go, and what are the trade-offs between the options?"

What a strong answer should cover:

  • Definition: a stateless service keeps nothing about a client in process memory between requests, so any instance can serve any request and instances can be added, killed or replaced freely
  • What breaks with in-memory sessions: round-robin sends the next request to an instance that never saw the login; sticky sessions patch that but lose sessions when the instance dies and skew load
  • The two real answers: a shared session store (session ID in a cookie, data in Redis or a database โ€” central and revocable, one lookup per request) or a signed token such as a JWT (no lookup, but hard to revoke before it expires and sent on every request)
  • Token details: HMAC vs public-key signatures, short exp plus refresh tokens, a jti denylist for logout, HttpOnly/Secure/SameSite cookies, the roughly 4 KB per-cookie budget
  • The other hidden state to move out: uploaded files, local caches that must be consistent, scheduled jobs, rate-limit counters, open WebSocket connections

Clarifying questions expected:

  • Do we need instant logout or permission revocation (for example banking or admin roles)?
  • Is this browser traffic, mobile apps, service-to-service calls, or long-lived WebSockets?

Code / implementation expected: Optional โ€” sketching the session lookup or a token sign/verify step helps; the example simulates all four approaches.

  • stateless
  • sessions

Try it yourself

Sketch your approach before you read the solution. That is what the interview tests.

Stuck? Get a hint.

A short nudge toward the approach. It will not give the answer away.

Solution

The solution is hidden so you can try first

14 min read ยท 11 sections

Read the code

Where the session lives: memory, sticky, shared store, signed token
// Where does the session live? Four answers, simulated in one process.
// "Instances" are objects, the "load balancer" is a function, the shared
// store is a Map standing in for Redis. No network, no real HTTP.
const crypto = require("crypto");

function makeCluster() {
  return [1, 2, 3].map((n) => ({ name: "inst-" + n, up: true, memory: new Map() }));
}
let counter = 0;
const newId = () => "sess-" + ++counter;

function roundRobin(cluster) {
  let i = 0;
  return () => {
    for (;;) { const inst = cluster[i++ % cluster.length]; if (inst.up) return inst; }
  };
}
function crash(inst) { inst.up = false; inst.memory.clear(); }

console.log("A) Session in instance memory, round-robin LB");
let cluster = makeCluster(), pick = roundRobin(cluster);
let inst = pick(), sid = newId();
inst.memory.set(sid, { user: "alice" });
console.log("  login  -> " + inst.name + " stores " + sid);
for (let i = 0; i < 3; i++) {
  inst = pick();
  console.log("  /cart  -> " + inst.name + (inst.memory.has(sid) ? " 200 alice" : " 401 who are you?"));
}

console.log("B) Session in instance memory, sticky LB");
cluster = makeCluster(); pick = roundRobin(cluster);
const sticky = new Map(); // session id -> instance (what a sticky cookie encodes)
const route = (id) => { const s = sticky.get(id); if (s && s.up) return s; const n = pick(); sticky.set(id, n); return n; };
sid = newId();
inst = route(sid); inst.memory.set(sid, { user: "alice" });
console.log("  login  -> " + inst.name + " stores " + sid);
inst = route(sid); console.log("  /cart  -> " + inst.name + (inst.memory.has(sid) ? " 200 alice" : " 401"));
crash(cluster[0]); console.log("  inst-1 crashes (deploy, OOM, scale-in)");
inst = route(sid); console.log("  /cart  -> " + inst.name + (inst.memory.has(sid) ? " 200 alice" : " 401 session lost"));

console.log("C) Session in a shared store, any instance");
cluster = makeCluster(); pick = roundRobin(cluster);
const store = new Map(); // stands in for Redis: SET sid value EX ttl
sid = newId(); inst = pick(); store.set(sid, { user: "alice" });
console.log("  login  -> " + inst.name + " writes " + sid + " to the store");
crash(cluster[0]); console.log("  inst-1 crashes");
for (let i = 0; i < 2; i++) {
  inst = pick();
  console.log("  /cart  -> " + inst.name + (store.has(sid) ? " 200 " + store.get(sid).user : " 401"));
}
store.delete(sid); inst = pick();
console.log("  logout deletes the key; /cart -> " + inst.name + (store.has(sid) ? " 200" : " 401"));

console.log("D) Signed token (JWT-shaped, HS256), nothing stored server-side");
const SECRET = "demo-secret-shared-by-all-instances";
const b64 = (s) => Buffer.from(s).toString("base64url");
const sign = (data) => crypto.createHmac("sha256", SECRET).update(data).digest("base64url");
function issue(claims) {
  const body = b64(JSON.stringify({ alg: "HS256", typ: "JWT" })) + "." + b64(JSON.stringify(claims));
  return body + "." + sign(body);
}
const denylist = new Set();
function verify(token, now) {
  const [h, p, sig] = token.split(".");
  const good = Buffer.from(sign(h + "." + p)), got = Buffer.from(sig);
  if (good.length !== got.length || !crypto.timingSafeEqual(good, got)) return "401 bad signature";
  const claims = JSON.parse(Buffer.from(p, "base64url").toString());
  if (now >= claims.exp) return "401 expired";
  if (denylist.has(claims.jti)) return "401 revoked";
  return "200 " + claims.sub + " (" + claims.role + ")";
}
const token = issue({ sub: "alice", role: "user", exp: 1900, jti: "t-1" });
console.log("  token: " + token);
console.log("  bytes in every request: " + token.length);
console.log("  inst-2 verifies at t=1000: " + verify(token, 1000));
const [h, , s] = token.split(".");
const forged = h + "." + b64(JSON.stringify({ sub: "alice", role: "admin", exp: 1900, jti: "t-1" })) + "." + s;
console.log("  role changed to admin:    " + verify(forged, 1000));
console.log("  user logs out at t=1100, token still: " + verify(token, 1100));
denylist.add("t-1");
console.log("  with jti on a denylist:   " + verify(token, 1100));
denylist.clear();
console.log("  at t=1900 (exp reached):  " + verify(token, 1900));

Discussion

Other approaches, follow-ups and how it went in real interviews.

Sign in to join the discussion.

No comments yet. Share how you would answer it.

Question 32 of 106 in System design

Keep your momentum going.

Mark this question solved to track your progress, then take the next one while it is fresh.

Up nextMedium

What is database replication?

Continue