Reranking in RAG: when and how to add a cross-encoder?

Q: Reranking in RAG: when and how to add a cross-encoder?

**Target Audience:** Engineers who already understand what a reranker is and now have to decide whether to put one into a real RAG system, which one, how deep, and inside what latency budget. **Difficulty:** Medium **Verified with:** Node 22.22.0 and Python 3.12.3 (the depth sweep and timeout simulation below). The relevance scores are simulated with a seeded generator and the reranker latency comes from a stated cost model, so the numbers show the *method*, not a benchmark of any product. Vendor API facts are cited inline with the date checked (October 2026). > **How to read this doc:** Every idea is explained in plain language first, then tagged with `📌 Interview term:` so you also learn the words an interviewer expects. All program output shown was produced by actually running the example; the **Verified** section quotes it in full. ## 1. Why This Even Matters — A Story First A restaurant kitchen keeps getting complaints that dishes are "almost right". The head chef could hire a

The problem

Question presented to candidate: "You own a RAG search feature with hybrid retrieval. Product says answers are often close but not quite right. How would you decide whether a cross-encoder reranker is the fix, and if it is, how would you add one: which model, how many candidates, and what happens to latency?"

What a strong answer should cover:

Diagnose first with a labelled eval set: if the answer chunk is usually in the top 50 to 100 but not in the top few you send, a reranker helps; if it is not in the shortlist at all, reranking cannot fix it and the work is in recall
Pick the rerank depth by measurement: a sweep of candidate depth against recall at that depth, nDCG or MRR after reranking, and reranker latency; quality flattens while latency grows linearly with pairs
Choosing the model: hosted rerank API versus self-hosted open cross-encoder, trading cost per call, network hop and data residency against running GPUs or CPUs yourself; check language coverage and the per-document token limit
Production wiring: batch the pairs in one call, rerank the same text that goes in the prompt (title plus chunk), set a time budget with a fallback to first-stage order, and threshold scores so low-relevance chunks are dropped
Keep evaluating after launch: log scores and positions, re-tune the floor and depth when the model, the corpus or the chunking changes

Clarifying questions expected:

What is the latency budget for the whole request, and how much is already spent before generation?
How large and how multilingual is the corpus, and can data leave our network for a hosted API?
Do we have labelled queries, or do we need to build a small eval set first?

Code / implementation expected: Optional — a retrieve-then-rerank function with a timeout fallback and a score floor, or the measurement plan for choosing depth.

rerank
cross-encoder

Try it yourself

Sketch your approach before you read the solution. That is what the interview tests.

Stuck? Get a hint.

A short nudge toward the approach. It will not give the answer away.

Solution

Target Audience: Engineers who already understand what a reranker is and now have to decide whether to put one into a real RAG system, which one, how deep, and inside what latency budget. Difficulty: Medium Verified with: Node 22.22.0 and Python 3.12.3 (the depth sweep and timeout simulation below). The relevance scores are simulated with a seeded generator and the reranker latency comes from a stated cost model, so the numbers show the method, not a benchmark of any product. Vendor API facts are cited inline with the date checked (October 2026).

How to read this doc: Every idea is explained in plain language first, then tagged with 📌 Interview term: so you also learn the words an interviewer expects. All program output shown was produced by actually running the example; the Verified section quotes it in full.

1. Why This Even Matters — A Story First

A re

The solution is hidden so you can try first

12 min read · 12 sections

Read the code

Rerank depth sweep with a recall ceiling, a latency model and a timeout fallback

Open in playground

// Choosing rerank depth with an eval set: recall ceiling, nDCG@5, latency budget, timeout fallback.
let seed = 20251;
function rand() { seed = (seed * 48271) % 2147483647; return seed / 2147483647; }
const noise = () => rand() + rand() + rand(); // 0..3, bell-shaped, no Math.log needed
const f3 = (x) => (Math.floor(x * 1000 + 0.5) / 1000).toFixed(3);

// 40 eval queries, 400 candidate chunks each. Grade 2 = answers it, 1 = partly, 0 = not.
const QUERIES = [];
for (let q = 0; q < 40; q++) {
  const docs = [];
  for (let d = 0; d < 400; d++) {
    const grade = d === 0 ? 2 : d < 3 ? 1 : 0;
    const vector = 0.75 * grade + noise();       // stage 1: one vector per side, lossy
    const cross = 1.2 * grade + 0.9 * noise();  // stage 2: reads the pair, sharper
    docs.push({ d, grade, vector, cross });
  }
  QUERIES.push(docs);
}
const byVector = (a, b) => b.vector - a.vector || a.d - b.d;
const byCross = (a, b) => b.cross - a.cross || a.d - b.d;
function dcg(list) { let s = 0; list.forEach((x, i) => { s += (2 ** x.grade - 1) / Math.log2(i + 2); }); return s; }
const IDEAL = dcg([{ grade: 2 }, { grade: 1 }, { grade: 1 }]);
const ndcg5 = (list) => dcg(list.slice(0, 5)) / IDEAL;
const mean = (xs) => xs.reduce((s, x) => s + x, 0) / xs.length;

// Assumed cost model for a self-hosted cross-encoder: 20 ms fixed + 1.5 ms per pair.
const rerankMs = (n) => 20 + 1.5 * n;

console.log("1) depth sweep over 40 labelled queries");
console.log("   depth  gold@depth  recall@depth  nDCG@5 no rerank  nDCG@5 reranked  rerank ms");
const base = mean(QUERIES.map((docs) => ndcg5([...docs].sort(byVector))));
for (const depth of [5, 10, 25, 50, 100, 200]) {
  const gold = [], recall = [], nd = [];
  for (const docs of QUERIES) {
    const cand = [...docs].sort(byVector).slice(0, depth);
    gold.push(cand.some((x) => x.grade === 2) ? 1 : 0);
    recall.push(cand.filter((x) => x.grade > 0).length / 3);
    nd.push(ndcg5([...cand].sort(byCross)));
  }
  console.log(`   ${String(depth).padStart(5)}  ${f3(mean(gold)).padStart(10)}  ${f3(mean(recall)).padStart(12)}  ${f3(base).padStart(16)}  ${f3(mean(nd)).padStart(15)}  ${String(rerankMs(depth)).padStart(9)}`);
}

console.log("");
console.log("2) budget 120 ms for reranking, depth 50, with a timeout fallback");
const DEPTH = 50, BUDGET = 120;
const loads = [1.0, 1.1, 0.9, 1.3, 1.0, 2.4, 1.0, 1.2, 0.95, 3.0]; // slowdown factor per request
let fellBack = 0;
loads.forEach((load, i) => {
  const need = rerankMs(DEPTH) * load;
  const ok = need <= BUDGET;
  const ms = ok ? need : BUDGET; // the call is abandoned at the budget
  if (!ok) fellBack++;
  const docs = QUERIES[i];
  const cand = [...docs].sort(byVector).slice(0, DEPTH);
  const top = ok ? [...cand].sort(byCross) : cand; // fallback: keep stage 1 order
  const pos = top.findIndex((x) => x.grade === 2);
  const where = pos < 0 ? "not in top 50" : pos < 5 ? `rank ${pos + 1}, sent` : `rank ${pos + 1}, cut`;
  console.log(`   req ${String(i + 1).padStart(2)}  rerank ${f3(ms).padStart(7)} ms  ${ok ? "reranked              " : "timeout, stage 1 order"}  gold: ${where}`);
});
console.log(`   fallbacks: ${fellBack} of ${loads.length}; every request still answered`);

Discussion

Other approaches, follow-ups and how it went in real interviews.

Sign in to join the discussion.

No comments yet. Share how you would answer it.

The solution is hidden so you can try first

Keep your momentum going.