Skip to the solution
MediumAI Engineering14 min read

When do you use code-based graders versus model-based graders?

969 views

The problem

Question presented to candidate: "You are setting up evals for an assistant that answers refund questions with a JSON record and a short customer message. Which checks would you write as code, which would you hand to an LLM grader, and how would you decide in general?"

What a strong answer should cover:

  • Code-based graders for anything with an objective answer: schema and parse checks, exact or normalised match, numeric tolerance, regex or forbidden-content rules, citations that must resolve, unit tests for generated code; they are cheap, fast, deterministic and debuggable
  • Model-based graders (LLM-as-judge) only for qualities code cannot decide: tone, faithfulness to sources, helpfulness, whether a free-text answer addresses the question; they need a rubric, cost a model call per case and carry biases
  • Decompose a vague quality into checkable parts first; often most of "is this answer good" turns into code checks, with one or two judged dimensions left
  • Validate any model grader against human labels before trusting it: confusion matrix, how many real failures it catches, kappa, and re-validation when the judge model or prompt changes
  • Layer them: code checks gate first (fail fast, no judge cost on broken output), judges on what survives, humans to calibrate both and to review disagreements

Clarifying questions expected:

  • Is there a single correct answer for parts of the output (IDs, amounts, policy windows), or is it all open-ended prose?
  • How often will the suite run, and what is the budget per run?
  • Do we have human-labelled examples to validate a judge against?

Code / implementation expected: Optional — a few code graders plus a stub judge over the same outputs, or a confusion matrix comparing a judge with human labels.

  • graders
  • evals
  • code-graders
  • llm-as-judge

Try it yourself

Sketch your approach before you read the solution. That is what the interview tests.

Stuck? Get a hint.

A short nudge toward the approach. It will not give the answer away.

Solution

The solution is hidden so you can try first

14 min read · 14 sections

Read the code

Five code graders and a stub judge on ten refund replies
Open in playground
// A grader stack for one task: reply to a refund request with JSON + a short message.
// Code graders check what is mechanical; a stubbed judge (canned verdicts standing in
// for model calls) checks tone and faithfulness. Human labels are the ground truth.
const POLICY = { P2: "Refunds within 30 days, paid in 5-7 business days.", P3: "Damaged items are replaced free." };
const GOOD = '{"order_id":1042,"amount":59.99,"currency":"USD","eligible":true}';
const outputs = [
  { id: 1, json: GOOD, reply: "Your refund of 59.99 USD is approved and takes 5-7 business days [P2].", human: "pass" },
  { id: 2, json: '{"order_id":1042,"amount":59.99,}', reply: "Refund approved, 5-7 business days [P2].", human: "fail" },
  { id: 3, json: GOOD.replace("59.99", "5.99"), reply: "Refund of 5.99 USD, 5-7 business days [P2].", human: "fail" },
  { id: 4, json: GOOD.replace("USD", "usd"), reply: "Refund of 59.99 USD approved, 5-7 business days [P2].", human: "pass" },
  { id: 5, json: GOOD, reply: "Guaranteed: your 59.99 USD will be back tomorrow, 5-7 business days at most [P2].", human: "fail" },
  { id: 6, json: GOOD, reply: "Refund of 59.99 USD in 5-7 business days [P9].", human: "fail" },
  { id: 7, json: GOOD, reply: "Read the policy next time. Refund of 59.99 USD in 5-7 business days [P2].", human: "fail" },
  { id: 8, json: GOOD, reply: "Refund of 59.99 USD in 5-7 business days [P2]. You also get a 20% coupon [P2].", human: "fail" },
  { id: 9, json: GOOD, reply: "Sorry for the trouble! Your refund of 59.99 USD is approved and arrives in 5-7 business days [P2].", human: "pass" },
  { id: 10, json: GOOD, reply: "Refund of 59.99 USD approved, it takes 5-7 days [P2].", human: "fail" },
];
// canned judge verdicts: what a rubric-following model judge returned for each case
const judge = {
  tone: { 7: "fail" },
  faithful: { 8: "fail" }, // note: judge was lenient on case 10 ("5-7 days" read as fine)
};

const code = {
  json_schema: (o) => {
    try { const j = JSON.parse(o.json); return ["order_id", "amount", "currency", "eligible"].every((k) => k in j); }
    catch { return false; }
  },
  fields: (o) => {
    try {
      const j = JSON.parse(o.json);
      return j.order_id === 1042 && Math.abs(j.amount - 59.99) < 0.005 && String(j.currency).toUpperCase() === "USD" && j.eligible === true;
    } catch { return false; }
  },
  window: (o) => o.reply.includes("5-7 business days"),
  no_promise: (o) => !/\b(tomorrow|today|guarantee[d]?)\b/i.test(o.reply),
  citations: (o) => { const c = o.reply.match(/\[P\d+\]/g) || []; return c.length > 0 && c.every((x) => x.slice(1, -1) in POLICY); },
};
const model = {
  judge_tone: (o) => judge.tone[o.id] !== "fail",
  judge_faith: (o) => judge.faithful[o.id] !== "fail",
};
const names = Object.keys(code).concat(Object.keys(model));
const all = { ...code, ...model };

console.log("case  " + names.map((n) => n.padEnd(13)).join("") + "human");
let codeOk = 0, bothOk = 0;
const byCode = [], byJudge = [];
for (const o of outputs) {
  const r = names.map((n) => (all[n](o) ? "ok" : "X"));
  const codeV = Object.keys(code).every((n) => code[n](o)) ? "pass" : "fail";
  const bothV = names.every((n) => all[n](o)) ? "pass" : "fail";
  if (codeV === "fail") byCode.push(o.id);
  if (!Object.keys(model).every((n) => model[n](o))) byJudge.push(o.id);
  if (codeV === o.human) codeOk++;
  if (bothV === o.human) bothOk++;
  console.log(String(o.id).padEnd(6) + r.map((x) => x.padEnd(13)).join("") + o.human);
}
console.log("");
console.log(`failures caught by a code grader: ${byCode.join(", ")}`);
console.log(`failures caught only by the judge: ${byJudge.join(", ")}`);
console.log(`code graders alone agree with humans on ${codeOk}/${outputs.length}`);
console.log(`code + judge agree with humans on ${bothOk}/${outputs.length}`);
console.log("");
// cost per full run, all numbers are assumptions for the arithmetic
const cases = 2000, dims = Object.keys(model).length, inTok = 900, outTok = 150, inPrice = 3, outPrice = 15;
const calls = cases * dims;
const usd = (calls * inTok * inPrice + calls * outTok * outPrice) / 1e6;
console.log(`assumed: ${cases} cases, ${dims} judge dimensions, ${inTok} in + ${outTok} out tokens per call, $${inPrice}/$${outPrice} per M tokens`);
console.log(`code graders: 0 model calls, $0.00 per run`);
console.log(`judge graders: ${calls} model calls, $${usd.toFixed(2)} per run, $${(usd * 30).toFixed(2)} per month at one run a day`);
Validating a model grader against human labels, and correcting its pass rate
Open in playground
// Before trusting a model-based grader, measure it against human labels.
// 50 human-labelled outputs; the judge verdicts are canned (standing in for model calls).
function build(humanPass, judgePassGivenPass, humanFail, judgeFailGivenFail) {
  const rows = [];
  for (let i = 0; i < humanPass; i++) rows.push({ human: "pass", judge: i < judgePassGivenPass ? "pass" : "fail" });
  for (let i = 0; i < humanFail; i++) rows.push({ human: "fail", judge: i < judgeFailGivenFail ? "fail" : "pass" });
  return rows;
}
function stats(rows) {
  const c = { pp: 0, pf: 0, fp: 0, ff: 0 };
  for (const r of rows) c[r.human[0] + r.judge[0]]++;
  const n = rows.length;
  const tpr = c.pp / (c.pp + c.pf); // judge says pass when humans say pass
  const tnr = c.ff / (c.ff + c.fp); // judge says fail when humans say fail
  const po = (c.pp + c.ff) / n;
  const hP = (c.pp + c.pf) / n, jP = (c.pp + c.fp) / n;
  const pe = hP * jP + (1 - hP) * (1 - jP);
  return { c, n, tpr, tnr, acc: po, kappa: (po - pe) / (1 - pe), humanRate: hP, judgeRate: jP };
}
const pct = (x) => (100 * x).toFixed(1) + "%";
function show(name, s) {
  console.log(name);
  console.log("                 judge pass  judge fail");
  console.log(`  human pass     ${String(s.c.pp).padStart(10)}  ${String(s.c.pf).padStart(10)}`);
  console.log(`  human fail     ${String(s.c.fp).padStart(10)}  ${String(s.c.ff).padStart(10)}`);
  console.log(`  accuracy ${pct(s.acc)}  kappa ${s.kappa.toFixed(2)}  catches ${pct(s.tnr)} of real failures  keeps ${pct(s.tpr)} of real passes`);
  console.log(`  pass rate: humans ${pct(s.humanRate)}, judge ${pct(s.judgeRate)}`);
}
const v1 = stats(build(35, 33, 15, 9));
show("judge v1: one 1-10 helpfulness score, pass if >= 7", v1);
console.log("");
const v2 = stats(build(35, 32, 15, 13));
show("judge v2: binary verdict per rubric criterion, reasoning before verdict", v2);
console.log("");
// the same 1000 production outputs graded by each judge; correct each with its measured error rates (Rogan-Gladen)
console.log("same 1000 production outputs, raw judge pass rate -> corrected estimate");
for (const [name, s, passed] of [["v1", v1, 850], ["v2", v2, 760]]) {
  const observed = passed / 1000;
  const corrected = (observed + s.tnr - 1) / (s.tpr + s.tnr - 1);
  console.log(`  judge ${name}: ${pct(observed)} -> ${pct(Math.min(1, Math.max(0, corrected)))}`);
}

Discussion

Other approaches, follow-ups and how it went in real interviews.

Sign in to join the discussion.

No comments yet. Share how you would answer it.

Question 23 of 83 in AI engineering

Keep your momentum going.

Mark this question solved to track your progress, then take the next one while it is fresh.

All AI engineering questions
Up nextMedium

What are reasoning models and when should you use them?

Continue