Why RAG
RAG answers from your documents, not just from training
Retrieve the right passages, then generate the answer from them.
Ask a model about your refund policy and it can only guess from general knowledge: thirty days, three to five days, plausible and wrong. Retrieval augmented generation first searches your own documents, finds the two passages that answer the question, and puts them into the prompt. The model answers from them and cites them. Change the policy document, and the next answer changes too.
Why RAG · Going deeper
RAG, a long prompt or fine-tuning?
RAG for knowledge; fine-tuning for behaviour.
The two pipelines
Index the documents once; retrieve for every question
Index: split, embed, store. Query: embed, search, prompt, answer.
RAG is two pipelines. Indexing runs ahead of time. Take the documents. Split them into chunks. Turn each chunk into a vector with an embedding model. Store the vector with its text and metadata. Then for every question: embed the question with the same model, find the nearest chunks in the store, put them into the prompt with the question, and the model writes a cited answer.
The two pipelines · Going deeper
Managed RAG services and vector stores
Managed services run both pipelines; pick the store you already know.
Chunking
How you split documents decides what can be found
500-token chunks, 50 tokens of overlap: 27 chunks.
Search returns chunks, not documents, so how you cut matters. This handbook is twelve thousand tokens. Cut into five hundred token chunks, each overlapping the last by fifty, gives 27 chunks. The overlap means a sentence cut at a boundary still appears whole in one of them. Smaller chunks match precisely but lose context; bigger ones keep context but blur the match. Splitting on headings and sections usually beats a fixed size.
Vector search
Vector search finds the chunks closest in meaning
Cosine similarity ranks chunks by angle to the question; keep the top-k.
Each chunk is a vector, drawn here as an arrow. The question is a vector too. Chunks that mean similar things point in similar directions, so search measures the angle, with cosine similarity. Refund timing is closest, at 0.99. Returns of opened items next. Then faulty products. These top three go into the prompt. Careers points the other way, and is left out.
Vector search · Going deeper
How stores search millions of vectors fast
HNSW and IVF skip most vectors and accept rare misses.
Hybrid search and reranking
Combine keyword and vector search, then rerank
Hybrid search fused with reciprocal rank fusion; a reranker picks the final few.
Vector search is weak on exact strings like a part number. Keyword search finds them, but misses synonyms. So run both. Reciprocal rank fusion scores each chunk as one over sixty plus its rank, summed across the lists. The return form for that part is first on keywords and third on vectors, so it scores 0.0323, and comes out on top. A reranker then reads the question with each candidate and keeps the best few.
BM25 · Going deeper
BM25: rare words weigh most, and repeats count less each time
BM25 = rarity (IDF) × saturating term frequency, adjusted for length.
Grounding and citations
Tell the model to answer only from the sources, and cite them
Grounding: only the sources, a citation per claim, “I don’t know” otherwise.
The retrieved chunks go into the prompt with clear markers. The instructions: answer only from these sources, and cite each claim. If they do not cover it, say so. The first source, about refund timing. The second, about opened software. Then the question. The answer uses both, and each claim carries its source number, which the app turns into a link.
Evaluating RAG
Test retrieval and generation separately
Context recall for retrieval; faithfulness and relevance for the answer.
Keep a set of test questions, each with the passage that answers it. Then you can see where things break. For 6 of 40, the right passage never came back: a retrieval problem, fix chunking or search. For 4, it came back but the answer was still wrong: a generation problem, fix the prompt. The other 30 were right. Without that split, you tune the wrong half.
Retrieval metrics · Going deeper
Recall@k, precision@k, MRR and nDCG, worked on one result list
Recall@k caps answer quality; MRR and nDCG reward relevant chunks near the top.
Lesson complete
You finished “Retrieval-augmented generation”
Nicely done.