Question 1

What are evals and why does every LLM feature need them?

Accepted Answer

**Target Audience:** Anyone building on top of an LLM API who has never written an eval, and candidates who need to explain why "it looked good in the playground" is not a test.
**Difficulty:** Easy
**Verified with:** Node 22 and Python 3.12 (an eval harness over two stubbed prompt versions, and a sampling simulation with Wilson intervals). No model is called; the stub replies stand in for a real model. Vendor guidance is quoted with its source, checked October 2026.

> **How to read this doc:**

Question 2

When do you use code-based graders versus model-based graders?

Accepted Answer

**Target Audience:** Engineers writing their first eval suite who need to decide, check by check, whether to write a function or ask a model, and candidates who should be able to defend that choice.
**Difficulty:** Medium
**Verified with:** Node 22 and Python 3.12 (five code graders and a stub judge over ten outputs, and a judge-versus-human validation with corrected pass rates). Judge verdicts are canned stand-ins for model calls; token prices in the cost line are assumptions. Vendor tooling is

Question 3

Beyond Ragas: how do you eval LLMs with LLM-as-judge, DeepEval, and human eval?

Accepted Answer

**Target Audience:** Engineers who have one eval framework running (often Ragas on a RAG endpoint) and now need to evaluate the rest of an LLM product: summaries, support replies, agents, anything without a neat retrieval score.
**Difficulty:** Medium
**Verified with:** Node 22.22.0 and Python 3.12.3 (the judge-calibration and review-routing example below). DeepEval and G-Eval details were checked against the DeepEval docs, PyPI and the G-Eval paper via web search in October 2026; where only sec

Question 4

What is LLM-as-judge and what are its known biases?

Accepted Answer

**Target Audience:** Engineers about to let a model grade their model, and candidates who need to name the biases and the fixes precisely.
**Difficulty:** Medium
**Verified with:** Node 22 and Python 3.12 (a pairwise judge with a simulated position bias, and a scoring judge with a simulated verbosity bias). The judges are deterministic stubs with a bias built in on purpose, so the mechanism and the fix are visible; the size of the bias is an assumption, not a measurement of any model. Published

Question 5

Why are BLEU and ROUGE poor metrics for LLM outputs, and what replaced them?

Accepted Answer

**Target Audience:** Engineers who have seen BLEU or ROUGE in a dashboard or a paper and need to explain why they mislead for LLM features, and candidates expected to know the modern metric stack.
**Difficulty:** Hard
**Verified with:** Node 22 and Python 3.12 (BLEU, ROUGE-1 and ROUGE-L implemented from scratch, scored on hand-written answers next to a key-fact check, then against different references). Published results are cited by paper, checked October 2026.

> **How to read this doc:** Ever

AI ENGINEERINGDECODED.

Easy

What are evals and why does every LLM feature need them?

Medium

When do you use code-based graders versus model-based graders?

Beyond Ragas: how do you eval LLMs with LLM-as-judge, DeepEval, and human eval?

What is LLM-as-judge and what are its known biases?

Hard

Why are BLEU and ROUGE poor metrics for LLM outputs, and what replaced them?