ALLMAcademy

D3

Retrieval Metrics, Index Freshness & Grounding Quality

12 min
  1. 01Compute recall@k, precision@k, MRR, and nDCG and explain why recall@k is the system ceiling
  2. 02Diagnose freshness failures: stale vectors, version skew, missing deletions, recency
  3. 03Evaluate grounding/faithfulness and citation precision vs. recall
  4. 04Explain why grounded, correct, and well-cited are three independent failure axes
Narration — Retrieval Metrics, Index Freshness & Grounding Quality
0:00 / 0:00

Measure retrieval separately

topic 16

Four core metrics, computed against labeled relevant docs:

  • recall@k — fraction of all relevant docs that appear in the top-k.
  • precision@k — fraction of the top-k that is actually relevant (noise / budget).
  • MRR — reciprocal rank of the first relevant hit.
  • nDCG — graded relevance, log-discounted by position, normalized to [0,1].

Raising k trades precision down for recall up.

Common MisconceptionA correct answer proves retrieval and grounding are good.

CorrectionThe model may be answering from parametric memory — the relevant doc may never have been retrieved at all.

The index is a cache that drifts

Treat the index as a versioned artifact

Incrementally embed new docs, re-embed edits, and tombstone deletions. Add recency via time-decay or date filters. Watch ANN params (e.g. HNSW ef) — set too low they silently drop neighbors and quietly lower recall. Swapping the embedding model forces a full re-index.

Grounding, correctness, and citation quality

Three independent questions

Grounding / faithfulness: is each claim entailed by the retrieved context (independent of whether it’s true)? Score by decomposing the answer into atomic claims and judging each with NLI or an LLM judge (RAGAS, TruLens RAG triad). Citation recall: are supported statements cited? Citation precision: do the cited passages actually support the claim (ALCE)?

Production failure modes

  • Recall ceiling hit → the generator confidently hallucinates a missing fact.
  • Deleted or updated docs still served because tombstoning lagged.
  • Citation hallucination: a passage is cited but doesn't actually support the claim (precision failure).
  • Eval run on non-representative synthetic questions, hiding real gaps.

Retrieval Practice

Check one idea at a time

Question 1 of 2

A fact exists in the corpus but the answer never states it. You measure precision@5 = 0.8 and recall@5 = 0.2. Which is the binding problem?