Question 1

What are evals and why does every LLM feature need them?

Accepted Answer

**Target Audience:** Anyone building on top of an LLM API who has never written an eval, and candidates who need to explain why "it looked good in the playground" is not a test.
**Difficulty:** Easy
**Verified with:** Node 22 and Python 3.12 (an eval harness over two stubbed prompt versions, and a sampling simulation with Wilson intervals). No model is called; the stub replies stand in for a real model. Vendor guidance is quoted with its source, checked October 2026.

> **How to read this doc:**

Question 2

AI SDK 5: when to use streamText vs generateText vs generateObject?

Accepted Answer

**Target Audience:** TypeScript developers starting with the Vercel AI SDK who want a clear rule for which core function to call.
**Difficulty:** Easy
**Verified with:** Node 22.22.0 and Python 3.12.3 for the simulation below (identical output). AI SDK facts were read in October 2026 from the published packages on npm: `ai@5.0.277` (latest 5.x) type definitions and source, and the migration guides shipped inside `ai@7.0.143`; release dates are from the npm registry.

> **How to read this doc:**

Question 3

How do you choose the right model for a task?

Accepted Answer

**Target Audience:** Engineers adding their first LLM feature who see a long list of models and want a repeatable way to pick one.
**Difficulty:** Easy
**Verified with:** Node 22.22.0 and Python 3.12.3 (the selection harness below; the models in it are deterministic stubs and the prices are illustrative). Product and paper facts are cited inline with the date they were checked (October 2026).

> **How to read this doc:** Every idea is explained in plain language first, then tagged with `📌 Inter

Question 4

How do you get reliable structured JSON output from an LLM?

Accepted Answer

**Target Audience:** Engineers whose LLM feature has to hand JSON to ordinary code (a parser, a database, an API) and who want it to stop breaking.
**Difficulty:** Medium
**Verified with:** Node 22.22.0 and Python 3.12.3 (the constrained-decoding simulation below). Provider API shapes were checked against the vendor docs and announcements cited inline, in October 2026.

> **How to read this doc:** Every idea is explained in plain language first, then tagged with `📌 Interview term:` so you also

Question 5

How do you build a golden dataset for evaluating an LLM feature?

Accepted Answer

**Target Audience:** Engineers who know why evals matter and now have to assemble the dataset behind them, and candidates asked to design one in an interview.
**Difficulty:** Medium
**Verified with:** Node 22 and Python 3.12 (a dataset builder over 200 simulated traces, and an inter-annotator agreement calculation). The traces and labels are synthetic and stated as such; no model is called.

> **How to read this doc:** Every idea is explained in plain language first, then tagged with `📌 Intervi

Question 6

When do you use code-based graders versus model-based graders?

Accepted Answer

**Target Audience:** Engineers writing their first eval suite who need to decide, check by check, whether to write a function or ask a model, and candidates who should be able to defend that choice.
**Difficulty:** Medium
**Verified with:** Node 22 and Python 3.12 (five code graders and a stub judge over ten outputs, and a judge-versus-human validation with corrected pass rates). Judge verdicts are canned stand-ins for model calls; token prices in the cost line are assumptions. Vendor tooling is

Question 7

What is context engineering and how does it differ from prompt engineering?

Accepted Answer

**Target Audience:** Engineers moving from single prompts to RAG systems and agents, who need to explain what changes when "the prompt" becomes a pipeline.
**Difficulty:** Medium
**Verified with:** Node 22.22.0 and Python 3.12.3 (a context assembler run over simulated candidate items with given token counts; no real model is called). Origins of the term and published guidance are cited inline, checked October 2026.

> **How to read this doc:** Every idea is explained in plain language first, the

Question 8

How do you choose between open-weight models and proprietary API models?

Accepted Answer

**Target Audience:** Engineers and tech leads who have shipped on a model API and are asked "should we run our own model?"
**Difficulty:** Medium
**Verified with:** Node 22.22.0 and Python 3.12.3 (the break-even model below; every input is a labelled assumption, not a quote). Licence and product facts are cited inline with the source and the date checked (October 2026).

> **How to read this doc:** Every idea is explained in plain language first, then tagged with `📌 Interview term:` so you also

Question 9

What is LLM-as-judge and what are its known biases?

Accepted Answer

**Target Audience:** Engineers about to let a model grade their model, and candidates who need to name the biases and the fixes precisely.
**Difficulty:** Medium
**Verified with:** Node 22 and Python 3.12 (a pairwise judge with a simulated position bias, and a scoring judge with a simulated verbosity bias). The judges are deterministic stubs with a bias built in on purpose, so the mechanism and the fix are visible; the size of the bias is an assumption, not a measurement of any model. Published

Question 10

How do you catch regressions when changing a prompt or upgrading a model?

Accepted Answer

**Target Audience:** Engineers who own an LLM feature in production and must decide whether a prompt edit or a model upgrade is safe to ship.
**Difficulty:** Medium
**Verified with:** Node 22.22.0 and Python 3.12.3 (the regression gate below, with a seeded simulator). The promptfoo CI integration and the "Adding Error Bars to Evals" paper were checked via web search in October 2026.

> **How to read this doc:** Every idea is explained in plain language first, then tagged with `📌 Interview term:

AI ENGINEERINGDECODED.

Easy

What are evals and why does every LLM feature need them?

AI SDK 5: when to use streamText vs generateText vs generateObject?

How do you choose the right model for a task?

Medium

How do you get reliable structured JSON output from an LLM?

How do you build a golden dataset for evaluating an LLM feature?

When do you use code-based graders versus model-based graders?

What is context engineering and how does it differ from prompt engineering?

How do you choose between open-weight models and proprietary API models?

What is LLM-as-judge and what are its known biases?

How do you catch regressions when changing a prompt or upgrading a model?

Hard

How do you evaluate a RAG pipeline end to end?

How do you evaluate and debug agent trajectories?

Why are BLEU and ROUGE poor metrics for LLM outputs, and what replaced them?

How do you version, test and safely roll out prompt changes?

What techniques reduce LLM latency at each stage of a request?

What is red-teaming an LLM application and how is it done?

How do agents maintain memory across steps and sessions?