MediumAI Engineering16 min read

What should you log and trace in an LLM application?

Q: What should you log and trace in an LLM application?

**Target Audience:** Engineers building their first production LLM feature who need to decide what goes into logs, metrics and traces; also interview candidates who will be asked "how would you debug a bad answer?". **Difficulty:** Medium **Verified with:** Node 22.22.0 and Python 3.12.3 (the log-record builder below, identical output in both). Prices in the example are made-up round numbers, not any provider's price list. > **How to read this doc:** Every idea is explained in plain language first, then tagged with `📌 Interview term:` so you also learn the words an interviewer expects. All program output shown was produced by actually running the example; the **Verified** section quotes it in full. ## 1. Why This Even Matters — A Story First A plane's flight recorder does not record "the flight went fine" or "the flight went badly". It records altitude, speed, every control input and the cockpit voices, second by second, so that when something goes wrong investigators can replay ex

42 views

61 of 83

The problem

Question presented to candidate: "You are about to launch a support chatbot with retrieval and a couple of tools. A week after launch a customer complains that the bot told them something wrong. What would you need to have logged to explain exactly what happened — and what should you deliberately not log?"

What a strong answer should cover:

A per-request record that makes any answer reproducible: trace ID, feature, prompt name, version and hash, requested and served model, parameters, retrieved document IDs with scores, every tool call with arguments, status and latency, token usage including cached tokens, cost, time to first token and total latency, finish reason, guardrail results, errors, and the (redacted) input and output
Why LLM apps need more than classic request logs: failures are silent (a wrong answer is a 200 OK), so the content and the context that produced it are the debugging evidence
Aggregated metrics with low-cardinality labels (feature, prompt version, model): p95 latency, error and truncation rates, cache hit rate, cost per feature, refusal and guardrail rates
Privacy by design: redact or tokenise PII before the record is written, hash user IDs with a keyed hash, limit retention and access, and keep raw content out of metrics
Feedback joined by trace ID, so a thumbs-down becomes an eval candidate with the exact version and context attached

Code / implementation expected: Optional — a log record schema or a small function that builds one with redaction and cost.

observability
tracing
logging
llmops

Try it yourself

Sketch your approach before you read the solution. That is what the interview tests.

Stuck? Get a hint.

A short nudge toward the approach. It will not give the answer away.

Solution

Target Audience: Engineers building their first production LLM feature who need to decide what goes into logs, metrics and traces; also interview candidates who will be asked "how would you debug a bad answer?". Difficulty: Medium Verified with: Node 22.22.0 and Python 3.12.3 (the log-record builder below, identical output in both). Prices in the example are made-up round numbers, not any provider's price list.

How to read this doc: Every idea is explained in plain language first, then tagged with 📌 Interview term: so you also learn the words an interviewer expects. All program output shown was produced by actually running the example; the Verified section quotes it in full.

1. Why This Even Matters — A Story First

A plane's flight recorder does not record "the flight went fine" or "the flight went badly". It records altitude, speed, every control input and

The solution is hidden so you can try first

16 min read · 11 sections

Read the code

Structured LLM log record: redaction, cost from usage, roll-ups and feedback join

Open in playground

// One structured record per LLM request: redact before storing, derive cost from
// usage, keep versions and IDs, then roll up metrics and turn feedback into eval cases.
const crypto = require("crypto");
const KEY = "from-a-secret-store";   // HMAC key: user IDs stay joinable, not readable
const PRICE = { "model-small": [0.5, 2.0], "model-large": [3.0, 15.0] }; // toy $ per 1M input/output tokens

function redact(text) {
  return text
    .replace(/[\w.+-]+@[\w-]+\.[\w.]+/g, "[EMAIL]")
    .replace(/\b\d(?:[ -]?\d){12,15}\b/g, "[CARD]")
    .replace(/\+?\d[\d -]{8,}\d/g, "[PHONE]");
}
const clip = (s, n) => (s.length > n ? s.slice(0, n) + "..." : s);

function record(req) {
  const [inP, outP] = PRICE[req.model];
  const billedIn = req.usage.input - req.usage.cached;
  const micro = Math.round(billedIn * inP + req.usage.cached * inP * 0.1 + req.usage.output * outP);
  return {
    trace_id: req.trace,
    feature: req.feature,
    user: crypto.createHmac("sha256", KEY).update(req.user).digest("hex").slice(0, 16),
    prompt: { name: req.prompt[0], version: req.prompt[1], hash: req.prompt[2] },
    model: { requested: req.model, served: req.served, temperature: req.temperature },
    retrieval: { doc_ids: req.docs, top_score: req.topScore },
    tools: req.tools.map((t) => ({ name: t[0], args: redact(t[1]), status: t[2], ms: t[3] })),
    usage: { input_tokens: req.usage.input, cached_tokens: req.usage.cached, output_tokens: req.usage.output },
    cost_usd: (micro / 1e6).toFixed(6),
    latency_ms: { ttft: req.ttft, total: req.total },
    finish_reason: req.finish,
    guardrails: req.guards,
    error: req.error,
    input: clip(redact(req.input), 60),
    output: clip(redact(req.output), 60),
  };
}

const requests = [
  { trace: "t-001", feature: "support-chat", user: "alice@example.com", prompt: ["support", 7, "a1b2c3"], model: "model-large", served: "model-large-2026-08-01", temperature: 0.2, docs: ["kb-12", "kb-40"], topScore: 0.82, tools: [["lookup_order", '{"email":"alice@example.com","order":"A-991"}', "ok", 140]], usage: { input: 3200, cached: 2800, output: 180 }, ttft: 640, total: 2900, finish: "stop", guards: [], error: null, input: "My order A-991 has not arrived, email me at alice@example.com", output: "Order A-991 shipped on Monday and should arrive tomorrow." },
  { trace: "t-002", feature: "support-chat", user: "bob", prompt: ["support", 7, "a1b2c3"], model: "model-large", served: "model-large-2026-08-01", temperature: 0.2, docs: ["kb-03"], topScore: 0.41, tools: [], usage: { input: 2900, cached: 2800, output: 410 }, ttft: 700, total: 6100, finish: "length", guards: ["low_retrieval_score"], error: null, input: "Can I pay with card 4111 1111 1111 1111 or call +44 20 7946 0958?", output: "You can pay by card. Our support line is open from 9 to 5 and" },
  { trace: "t-003", feature: "summarize", user: "carol", prompt: ["summary", 3, "9f8e7d"], model: "model-small", served: "model-small-2026-05-20", temperature: 0, docs: [], topScore: null, tools: [], usage: { input: 5200, cached: 0, output: 300 }, ttft: 350, total: 1800, finish: "stop", guards: [], error: null, input: "Summarize this meeting transcript", output: "Three decisions were made: ship Friday, freeze the API, hire." },
  { trace: "t-004", feature: "support-chat", user: "dave", prompt: ["support", 7, "a1b2c3"], model: "model-large", served: null, temperature: 0.2, docs: [], topScore: null, tools: [], usage: { input: 0, cached: 0, output: 0 }, ttft: null, total: 30000, finish: null, guards: [], error: "timeout after 30000 ms", input: "Where is my refund?", output: "" },
];
const log = requests.map(record);
console.log("stored record for t-002:");
console.log(JSON.stringify(log[1], null, 2));

// Roll-ups for dashboards: low-cardinality labels only (feature, prompt version, model).
console.log("metrics by feature:");
for (const f of [...new Set(log.map((r) => r.feature))]) {
  const rs = log.filter((r) => r.feature === f);
  const lat = rs.map((r) => r.latency_ms.total).sort((a, b) => a - b);
  const p95 = lat[Math.ceil(0.95 * lat.length) - 1];
  const cost = rs.reduce((s, r) => s + Math.round(Number(r.cost_usd) * 1e6), 0) / 1e6;
  const inTok = rs.reduce((s, r) => s + r.usage.input_tokens, 0);
  const cached = rs.reduce((s, r) => s + r.usage.cached_tokens, 0);
  const errors = rs.filter((r) => r.error).length;
  const truncated = rs.filter((r) => r.finish_reason === "length").length;
  console.log(`  ${f}: n=${rs.length} p95=${p95}ms errors=${errors} truncated=${truncated} ` +
    `cache_hit=${inTok ? ((100 * cached) / inTok).toFixed(1) : "0.0"}% cost=$${cost.toFixed(6)}`);
}

// Feedback arrives later, keyed by trace_id: the join turns it into eval candidates.
const feedback = [["t-001", "up"], ["t-002", "down", "it cut off mid sentence"]];
console.log("eval candidates:");
for (const [tid, vote, note] of feedback) {
  if (vote !== "down") continue;
  const r = log.find((x) => x.trace_id === tid);
  console.log(`  ${tid} prompt=${r.prompt.name}@v${r.prompt.version} finish=${r.finish_reason} ` +
    `guards=${r.guardrails.join(",")} note="${note}"`);
  console.log(`    input: ${r.input}`);
}
const leaked = JSON.stringify(log).match(/alice@example\.com|4111|7946/g);
console.log(`raw PII strings left in the stored log: ${leaked ? leaked.length : 0}`);

Discussion

Other approaches, follow-ups and how it went in real interviews.

No comments yet. Share how you would answer it.

Question 61 of 83 in AI engineering

Keep your momentum going.

Mark this question solved to track your progress, then take the next one while it is fresh.

All AI engineering questions

Up nextHard→

How should an agent handle tool errors, timeouts and retries?

Continue