D3
Hybrid Search, RRF Fusion & Cross-Encoder Reranking
- 01Explain why dense and sparse retrieval fail on opposite query types and why hybrid is the default
- 02Describe Reciprocal Rank Fusion and why it fuses on ranks instead of normalizing scores
- 03Distinguish bi-encoder first-stage retrieval from cross-encoder reranking
- 04Reason about the K / latency / precision tradeoff in a two-stage pipeline
Dense and sparse cover each other's blind spots
topic 15Dense (bi-encoder) search captures meaning but under-weights exact tokens — IDs, error codes, SKUs, rare names. BM25 sparse/lexical search is the inverse: great on exact and rare terms, weak on paraphrase. Run both and fuse them; that’s hybrid search, the modern default. (SPLADE-style learned sparse models blur the line.)
The query fans into BM25 and dense pipes; RRF interleaves the ranked lists; the cross-encoder scores each survivor in one pass and the top 5–10 narrow into the LLM.
Text description
The query fans into BM25 and dense pipes; RRF interleaves the ranked lists; the cross-encoder scores each survivor in one pass and the top 5–10 narrow into the LLM.
Fusing with RRF
Reciprocal Rank Fusion scores each document as the sum over lists of 1 / (k + rank), with k ≈ 60 a common (tunable) default. Because it uses rank position, not raw similarity, it never has to reconcile a cosine score with a BM25 score.
Common MisconceptionJust average the dense and BM25 scores.
CorrectionThey live on different scales, so averaging is dominated by whichever scale is larger — fuse on ranks with RRF instead.
Common MisconceptionBM25 is obsolete now that embeddings exist.
CorrectionBM25 still wins on exact and rare terms; hybrid (dense + BM25) is the default, not a fallback.
The cross-encoder rerank stage
Two stages: cheap recall, then expensive precision
The first stage (ANN, often hybrid) fetches a wide top-K (~50–200). A cross-encoder then jointly encodes query + candidate, scores each, and keeps the best 5–10 — one forward pass per candidate, so its vectors can’t be precomputed.
| Raise K into the reranker | Effect |
|---|---|
| Larger K (e.g. 200) | Higher recall ceiling, but more cross-encoder passes → latency climbs |
| Smaller K (e.g. 50) | Faster reranking, but misses relevant docs the first stage ranked low |
Production failure modes
- Semantic-only retrieval silently missing exact identifiers like ERR_2049.
- Reranker latency blowup forcing K down until recall suffers.
- Expecting the reranker to surface a doc the first stage never retrieved.
Retrieval Practice
Check one idea at a time
Users search for the exact error code 'ERR_2049' and pure dense retrieval keeps missing the doc that contains it. Best fix?