Why RAG

RAG answers from your documents, not just from training

Question: Can I return opened software, and how long does a refund take?Model aloneMost retailers accept returns within 30days, and refunds take 3 to 5 days.✗general knowledge, no source, possibly inventedRetrieve, then answerrefund-policy.pdf, section 4 and section 2Only if it is faulty [2]. Refunds arrivewithin 14 days of receiving it [1].✓your facts, with sources a reader can checkRAG looks up your documents at question time and puts the relevant passages in the prompt.A model knows only what was in its training data, up to a cutoff, and nothing about your company. Retrieval-augmentedgeneration finds the passages that answer the question and adds them to the prompt, so the answer uses current, private factsand can cite them. The model is unchanged; update a document and the next answer uses it.

Retrieve the right passages, then generate the answer from them.

Why RAG · Going deeper

RAG, a long prompt or fine-tuning?

ApproachHowUse it forLong promptpaste the documents ina few documents; simple, costly per callRAGretrieve passages per questionlarge or changing knowledge, citationsFine-tuningtrain on examplesstyle, format, a narrow skill; not factsPretrainingtrain from scratcha new foundation model; rarely youUse RAG for knowledge that is large or changes; fine-tuning teaches behaviour, not facts.Even with context windows of a million tokens, sending everything on every call is slow and expensive, and models use themiddle of a very long prompt less reliably. Fine-tuning bakes facts in at training time, so they go stale and cannot be citedor removed per user. RAG keeps knowledge in a store you can update, filter by permission and audit. Lesson 12 compares thefull ladder of customisation.

RAG for knowledge; fine-tuning for behaviour.

The two pipelines

Index the documents once; retrieve for every question

Index: ahead of time, and again when documents changeDocumentsPDF, wiki, ticketsSplitinto chunksEmbedeach chunkVector storevectors + textQuery: on every questionQuestionfrom the userEmbedsame modelquery vectornearest chunksSearchtop-k chunksPromptquestion + chunksModelcited answerIndex and query must use the same embedding model: vectors from different models are not comparable.RAG is two pipelines: index the documents once, then retrieve and answer per question.Indexing splits documents into chunks, turns each into a vector and stores it with its text and metadata. Each question isembedded the same way, the store returns the nearest chunks, and the model answers from them.

Index: split, embed, store. Query: embed, search, prompt, answer.

The two pipelines · Going deeper

Managed RAG services and vector stores

ProviderManaged RAGVector storesAWSBedrock Knowledge BasesOpenSearch Serverless, Aurora pgvector, S3 Vectors, Neptune, partnersMicrosoftAzure AI Search, Foundry IQAzure AI Search index; Cosmos DB and PostgreSQL vector searchGoogleVertex AI RAG EngineVertex AI Vector Search, AlloyDB and Cloud SQL (pgvector)Open sourceLangChain, LlamaIndexpgvector, OpenSearch, FAISS, Milvus, Qdrant, ChromaManaged RAG services run both pipelines for you; the vector store is a choice underneath.A managed service connects to your sources (S3, SharePoint, websites), chunks and embeds them, keeps the index in sync, andanswers through one API call with citations. Pick the store by what you already run: a team on PostgreSQL can add pgvector;a team needing keyword plus vector search at scale picks a search engine. Store metadata (source, date, access group) withevery chunk so results can be filtered and permissions enforced.

Managed services run both pipelines; pick the store you already know.

Chunking

How you split documents decides what can be found

A 12,000-token handbook, 500-token chunks, 50-token overlap12,000 tokens123456789… 27 chunkseach chunk starts 450 tokens after the last: (12,000 − 50) ÷ 450 = 26.56, rounded up to 27the darker ends repeat the last 50 tokens of the previous chunk, so a sentence cut at a boundary survives (drawn 2.4× wider)Chunk size decides what can be found: small chunks are precise, large ones keep context.Too small, and a chunk loses the sentence that explains it; too large, and one chunk mixes topics and its vector matches nothingwell. A few hundred tokens with 10 to 20 percent overlap is a common start. Better still, split on structure: headings, sections,paragraphs. Hierarchical chunking searches small chunks but returns their parent section; semantic chunking splits where themeaning changes. Tune it on your own test questions.

500-token chunks, 50 tokens of overlap: 27 chunks.

Vector search

Vector search finds the chunks closest in meaning

Refund timingReturns: opened itemsFaulty productsDelivery timesStore opening hoursCareersGift cardsquestionCosine similarity to the questionRefund timing0.99Returns: opened items0.97Faulty products0.93Gift cards0.82Delivery times0.71Store opening hours0.34Careers-0.42top-3Cosine compares direction, not length: 1 = samemeaning, 0 = unrelated. Real embeddings havehundreds to thousands of dimensions, not two.Nearest by meaning, so "refund window" finds"money back within 14 days" with no shared words.

Cosine similarity ranks chunks by angle to the question; keep the top-k.

Vector search · Going deeper

How stores search millions of vectors fast

Layer 2Layer 1Layer 0: every vectornearest foundExact search compares the question with all 10,000,000 vectors. HNSW starts on a sparse top layer, jumps close,then refines on denser layers, visiting only a tiny fraction. IVF clusters vectors and searches the nearest clusters.Approximate nearest-neighbour indexes trade a little recall for a lot of speed.Real stores do not compare every vector; they use an index such as HNSW (a layered graph) or IVF (clusters). A query takesmilliseconds at millions of vectors, and now and then misses a true neighbour. Settings such as ef_search or the number ofclusters probed trade speed against recall. Metadata filters (date, team, access group) narrow the search further.

HNSW and IVF skip most vectors and accept rare misses.

Hybrid search and reranking

Combine keyword and vector search, then rerank

Query: "return form for part XR-4471B"ChunkKeyword rankVector rankRRF = Σ 1 ÷ (60 + rank)FusedReturn form XR-4471B130.0323#1Warranty terms410.0320#2Shipping faulty parts340.0315#3Part list, model XR270.0311#4Returning a faulty part–20.0161#5#1 is 0.0323 = 1 ÷ 61 + 1 ÷ 63: high on both lists beats first on only one.Hybrid search merges keyword and vector results; a reranker then reorders the best few.Vector search understands meaning but can miss exact strings: part numbers, error codes, names. Keyword search (BM25) findsthem but misses synonyms. Run both and fuse the ranked lists with reciprocal rank fusion, which needs only the ranks. A reranker,a model that reads the question and each candidate together, then reorders the top 20 or 50 and keeps the best 3 to 5.

Hybrid search fused with reciprocal rank fusion; a reranker picks the final few.

BM25 · Going deeper

BM25: rare words weigh most, and repeats count less each time

Query: "return XR-4471B" · k1 = 1.2, b = 0.75 (the Lucene and Elasticsearch defaults)Chunk"return" ×XR-4471B ×WordsBM25 scorereturn form for part XR-4471B115how to return a faulty part and get a return label2011warranty terms for every part005return policy: return within 30 days, return shipping is fre…40141.890.460.000.55Rarity: "return" is in 3 of 4 chunks, IDF 0.36. "XR-4471B" is in 1, IDF 1.20: 3.4 times the weight.Saturation: chunk 4 says "return" 4 times and earns 0.55 for it; chunk 1 says it once and earns 0.43. Four mentions are not four times the score.Length: scores are scaled by chunk length against the average (8.8 words), so a long chunk needs more matches.BM25 scores a chunk by its rare matching words, with repeats counting less each time.For each query word: IDF = ln(1 + (N − n + 0.5) ÷ (n + 0.5)), times tf × (k1 + 1) ÷ (tf + k1 × (1 − b + b × length ÷ average)).k1 sets how fast repeats saturate; b sets how much length is penalised. It needs no model and no GPU, and it finds the exactpart number that an embedding blurs. That is the keyword half of hybrid search. Invented chunks; scores computed here.

BM25 = rarity (IDF) × saturating term frequency, adjusted for length.

Grounding and citations

Tell the model to answer only from the sources, and cite them

System: Answer only from the sources. Cite each claim as [n].If the sources do not answer the question, say "I don’t know".<source id="1" doc="refund-policy.pdf §4">Refunds are issued within 14 days of receiving the return.</source><source id="2" doc="refund-policy.pdf §2">Opened software cannot be returned unless it is faulty.</source>Question: Can I return opened software, and how long does a refund take?AnswerOnly if it is faulty [2]. Refunds are issued within 14 days of receiving the return [1].each claim points to a source the app can show as a linkno source covers it? the model is told to say so rather than guessGrounding means answering only from the retrieved sources, with each claim cited.Tag each chunk with an id and its document, tell the model to use only those, to cite, and to say when the sources fallshort. The app turns [1] into a link, so a reader can check. Grounding lowers hallucination but does not remove it: measureit (next pages), and use contextual grounding checks (Bedrock Guardrails, Azure AI Content Safety groundedness detection).

Grounding: only the sources, a citation per claim, “I don’t know” otherwise.

Evaluating RAG

Test retrieval and generation separately

40 test questions, each with the passage that answers it and a reference answer6430Right chunk not retrieved: 6fix retrieval: chunking, hybrid search, reranker, top-kRetrieved, answer still wrong: 4fix generation: grounding instructions, prompt, modelRetrieved and answered correctly: 30Retrieval: right passage in the top 5 for 34/40 = 85% (context recall).Generation: correct when it had the passage, 30/34 = 88% (faithfulness and correctness).Score retrieval and generation separately, so you know which half to fix.Retrieval metrics: did the right passage come back (context recall, recall@k) and how much of what came back was useful (contextprecision). Generation metrics: is every claim supported by the passages (faithfulness, groundedness), does it answer the question(relevance), does it match the reference (correctness). Bedrock, Foundry and Vertex AI all offer RAG evaluations, often scored by asecond model as judge; spot-check the judge against people.

Context recall for retrieval; faithfulness and relevance for the answer.

Retrieval metrics · Going deeper

Recall@k, precision@k, MRR and nDCG, worked on one result list

One test question. Three chunks in the store answer it. The retriever returned these five, in order:rank 1not relevantrank 2relevantrank 3not relevantrank 4relevantrank 5not relevantthe third relevant chunkwas not retrieved at allMetricSumValueAsksPrecision@52 relevant ÷ 5 returned0.40how much is noise?Recall@52 found ÷ 3 that exist0.67was anything missed?Reciprocal rank1 ÷ rank of the first relevant = 1 ÷ 20.50how soon is the first hit?nDCG@5(0.631 + 0.431) ÷ (1 + 0.631 + 0.5) = 1.062 ÷ 2.1310.50are hits near the top?Four numbers describe a ranked list: how clean, how complete, how soon, and how well ordered.Average each over the whole test set; the mean reciprocal rank is MRR. Recall@k is the one that caps answer quality, becausea chunk that was never retrieved cannot be cited: raise k, add hybrid search or fix chunking. Low precision with good recall isa job for the reranker. nDCG discounts each hit by log₂(rank + 1) and divides by the best possible order, so 1.0 is a perfect list.

Recall@k caps answer quality; MRR and nDCG reward relevant chunks near the top.

Lesson complete

You finished “Retrieval-augmented generation”

Nicely done.

Open in a new tab