Skip to the solution
HardAI Engineering18 min read

What is quantization and when would you use a quantized model?

237 views

The problem

Question presented to candidate: "We want to self-host an open-weight model. Someone suggested running a 4-bit quantized version. Explain what quantization actually does to the model, why it makes inference cheaper and faster, what it costs in quality, and how you would decide whether to ship it."

What a strong answer should cover:

  • Quantization stores weights (and optionally activations or the KV cache) in fewer bits: map each value to a small integer grid with a scale factor, w ≈ q × scale; 16-bit to 4-bit cuts weight memory about 4x
  • Why it speeds things up: single-sequence decoding is memory-bandwidth bound (every weight is read once per generated token), so fewer bytes per weight raises the tokens-per-second ceiling, not just the file size
  • Granularity and outliers: one large value stretches the scale for everything that shares it; per-group scales (32 to 128 weights) and outlier-aware methods (LLM.int8(), AWQ, GPTQ) keep 4-bit error manageable at a small bits-per-weight overhead
  • The main formats and where they run: GGUF / llama.cpp for CPUs and consumer GPUs, GPTQ and AWQ for GPU serving, bitsandbytes NF4 (QLoRA) for fine-tuning on a quantized base
  • When to use it: fitting a bigger model on the hardware you have, on-device and local use, cutting serving cost; when not to: tasks where small accuracy losses matter (maths, code, long reasoning) without an eval on your own data
  • The decision rule: compare "bigger model, quantized" against "smaller model, full precision" on your own eval, measuring quality, latency and memory

Clarifying questions expected:

  • What hardware are we serving on (GPU memory, CPU only, edge device) and what batch sizes?
  • Which tasks matter most, and do we have an eval that would detect a few points of quality loss?

Code / implementation expected: Optional — a small absmax quantizer showing per-tensor vs per-group error, the outlier effect, and the memory and bandwidth arithmetic.

  • quantization
  • inference
  • open-weights
  • self-hosting

Try it yourself

Sketch your approach before you read the solution. That is what the interview tests.

Stuck? Get a hint.

A short nudge toward the approach. It will not give the answer away.

Solution

The solution is hidden so you can try first

18 min read · 11 sections

Read the code

Absmax INT8/INT4 quantization: per-tensor vs per-group error, one outlier, and memory and bandwidth arithmetic
Open in playground
// Weight quantization on a toy layer: absmax INT8 / INT4, per-tensor vs per-group,
// what one outlier does, and the memory and decode-speed arithmetic.
let seed = 2024;
const rand = () => { seed = (seed * 48271) % 2147483647; return seed / 2147483647; };
const gauss = () => { let s = 0; for (let i = 0; i < 12; i++) s += rand(); return (s - 6) * 0.02; }; // ~N(0, 0.02)
const N = 4096;
const w = Array.from({ length: N }, gauss);  // one row of "FP16" weights
const xs = Array.from({ length: 64 }, () => Array.from({ length: N }, () => rand() * 2 - 1)); // 64 activations
const roundHalfAway = (v) => Math.sign(v) * Math.floor(Math.abs(v) + 0.5); // same rule in both languages

// Symmetric absmax quantization: q = round(w / scale), scale = max|w| / qmax, w' = q * scale.
function quantize(weights, bits, group) {
  const qmax = 2 ** (bits - 1) - 1; // INT8 -> 127, INT4 -> 7
  const out = new Array(weights.length);
  for (let g = 0; g < weights.length; g += group) {
    const block = weights.slice(g, g + group);
    const scale = Math.max(...block.map(Math.abs)) / qmax;
    block.forEach((v, i) => { out[g + i] = Math.max(-qmax, Math.min(qmax, roundHalfAway(v / scale))) * scale; });
  }
  return out;
}
const dot = (a, b) => a.reduce((s, v, i) => s + v * b[i], 0);
const pct = (v) => (100 * v).toFixed(2).padStart(6) + "%";
function report(name, weights, bits, group) {
  const q = quantize(weights, bits, group);
  let dw = 0, nw = 0, dy = 0, ny = 0;
  weights.forEach((v, i) => { dw += (q[i] - v) ** 2; nw += v * v; });
  for (const x of xs) { const e = dot(weights, x), a = dot(q, x); dy += (a - e) ** 2; ny += e * e; }
  const zeros = q.filter((v) => v === 0).length;
  const bpw = bits + (group < N ? 16 / group : 0); // plus one FP16 scale per group
  console.log(`${name.padEnd(16)} bits/weight=${bpw.toFixed(3)} weight err=${pct(Math.sqrt(dw / nw))} ` +
    `output err=${pct(Math.sqrt(dy / ny))} rounded to zero=${String(zeros).padStart(4)} of ${N}`);
}
console.log("max|w| =", Math.max(...w.map(Math.abs)).toFixed(4));
report("INT8 per-tensor", w, 8, N);
report("INT4 per-tensor", w, 4, N);
report("INT4 group=128", w, 4, 128);
report("INT4 group=32", w, 4, 32);

// One outlier weight (LLMs have them) stretches the scale for everything that shares it.
const wo = w.slice(); wo[100] = 0.8;
console.log("with one outlier (0.8) at index 100:");
report("INT8 per-tensor", wo, 8, N);
report("INT4 per-tensor", wo, 4, N);
report("INT4 group=32", wo, 4, 32);

// Weights only (KV cache and activations come on top). Decode reads every weight once per
// token, so tokens/s per sequence <= bandwidth / weight bytes (assumed 1000 GB/s GPU).
for (const P of [8e9, 70e9]) {
  console.log(`weights only, ${(P / 1e9).toFixed(0)}B-parameter model:`);
  for (const [fmt, bpw] of [["FP16/BF16", 16], ["INT8 (+scale/32)", 8.5], ["INT4 (+scale/32)", 4.5], ["INT4 (+scale/128)", 4.125]]) {
    const bytes = P * bpw / 8;
    console.log(`  ${fmt.padEnd(18)} ${(bytes / 1e9).toFixed(1).padStart(6)} GB | fits 24 GB: ${bytes < 24e9 ? "yes" : "no "} | fits 80 GB: ${bytes < 80e9 ? "yes" : "no "} | decode ceiling ~${Math.floor(1000e9 / bytes)} tokens/s`);
  }
}

// KV cache for an assumed 8B shape (32 layers, 8 KV heads, head size 128): K and V per token per layer.
const kvPerToken = 2 * 32 * 8 * 128; // values per token
for (const ctx of [8192, 131072]) {
  const cells = [["FP16", 2], ["INT8", 1]].map(([n, b]) => `${n} ${(kvPerToken * b * ctx / 1e9).toFixed(1)} GB`).join(" | ");
  console.log(`KV cache, one sequence of ${ctx.toLocaleString("en-US")} tokens: ${cells}`);
}

Discussion

Other approaches, follow-ups and how it went in real interviews.

Sign in to join the discussion.

No comments yet. Share how you would answer it.

Question 81 of 83 in AI engineering

Keep your momentum going.

Mark this question solved to track your progress, then take the next one while it is fresh.

All AI engineering questions
Up nextHard

What is red-teaming an LLM application and how is it done?

Continue