Question 1

What are evals and why does every LLM feature need them?

Accepted Answer

**Target Audience:** Anyone building on top of an LLM API who has never written an eval, and candidates who need to explain why "it looked good in the playground" is not a test.
**Difficulty:** Easy
**Verified with:** Node 22 and Python 3.12 (an eval harness over two stubbed prompt versions, and a sampling simulation with Wilson intervals). No model is called; the stub replies stand in for a real model. Vendor guidance is quoted with its source, checked October 2026.

> **How to read this doc:**

Question 2

How do you estimate and control LLM costs in production?

Accepted Answer

**Target Audience:** Engineers asked "what will this cost per month?" before an LLM launch, and anyone whose first invoice was a surprise.
**Difficulty:** Medium
**Verified with:** Node 22.22.0 and Python 3.12.3 (a cost estimator and a metering ledger). All prices in the examples are a **hypothetical price card** with round numbers so the arithmetic is easy to follow; they are not any vendor's current prices. Discount mechanics (cache read and write multipliers, batch discounts) are quoted from

Question 3

What should you log and trace in an LLM application?

Accepted Answer

**Target Audience:** Engineers building their first production LLM feature who need to decide what goes into logs, metrics and traces; also interview candidates who will be asked "how would you debug a bad answer?".
**Difficulty:** Medium
**Verified with:** Node 22.22.0 and Python 3.12.3 (the log-record builder below, identical output in both). Prices in the example are made-up round numbers, not any provider's price list.

> **How to read this doc:** Every idea is explained in plain language fi

Question 4

How do you version, test and safely roll out prompt changes?

Accepted Answer

**Target Audience:** Engineers who own an LLM feature in production and need prompt edits to ship as safely as code changes; also anyone preparing for an AI engineering or ML platform interview.
**Difficulty:** Hard
**Verified with:** Node 22.22.0 and Python 3.12.3 (the registry, gate and canary simulation below, identical output in both). Tool behaviour (Langfuse labels, model aliases) was checked with web searches in October 2026 and is cited inline.

> **How to read this doc:** Every idea is

AI ENGINEERINGDECODED.

Easy

What are evals and why does every LLM feature need them?

Medium

How do you estimate and control LLM costs in production?

What should you log and trace in an LLM application?

Hard

How do you version, test and safely roll out prompt changes?