D3
Chunking, Embeddings & Bi-Encoder Vector Search
- 01Explain how chunk size and overlap trade precision against recall and dilute the embedding signal
- 02Describe the bi-encoder pipeline: independent encoding, precomputed doc vectors, ANN index
- 03Justify why the same embedding model and version must be used for indexing and querying
- 04Choose a chunking strategy and explain why structure-aware splits beat fixed-length cuts
Chunking is a precision/recall knob
topic 14You can’t embed a whole document and expect sharp retrieval. You split it into chunks (~200–500 tokens is a common heuristic), embed each, and index the vectors.
- Oversized chunks blur several topics into one vector and waste context budget.
- Undersized chunks fragment a single idea across many vectors, hurting recall.
- Overlap (~10–20%) keeps an idea that straddles a boundary from being split.
- Structure-aware splits (headings, sentences, code AST) beat blind fixed-length cuts.
Documents are chunked and embedded into an ANN index offline; at query time the question is embedded once and its nearest neighbors enter the funnel that narrows toward the LLM.
Text description
Documents are chunked and embedded into an ANN index offline; at query time the question is embedded once and its nearest neighbors enter the funnel that narrows toward the LLM.
The bi-encoder pipeline
Encode independently, search at scale
A bi-encoder runs the query and each document through the model separately. Document vectors are computed once, stored in an ANN index (HNSW, IVF). At query time you embed only the question and fetch nearest neighbors by cosine/dot similarity.
Knobs: dimensionality (~384–3072), domain fit, max sequence length, and quantization (smaller/faster vs. some accuracy loss). Model names like text-embedding-3, Cohere, or BGE/E5 are examples, not prescriptions.
Common MisconceptionCosine similarity equals relevance.
CorrectionNearness is not answering — it under-weights exact tokens (IDs, error codes) and can rank a topically-near but unhelpful chunk first.
Common MisconceptionYou can swap the embedding model without re-indexing.
CorrectionVectors from a different model/version aren't comparable — the index and query path must share the exact same model.
Production failure modes
- Boundary split with no overlap: a key sentence is cut in half across two chunks.
- One chunking strategy forced onto every doc type (prose, tables, code).
- Embedding-model version skew between index and query path.
- Near-duplicate chunks crowding the top-k with redundant hits.
Retrieval Practice
Check one idea at a time
A single 1500-token chunk covering both 'installation' and 'troubleshooting' ranks poorly for a specific troubleshooting query, even though the answer is in it. Best explanation?