The problem

Judge an assistant by the answers it should not give

2,000 help articles30,000 questions a monthAssistantAnswer, citingthe articles usedHand off to a personwhen unsureRequirementsAnswer in under 5 sNever invent a featureor a priceCite every claimStay within a budgetAn assistant is judged on the answers it should not give as much as on the ones it gives.A software company has 2,000 help articles and receives about 30,000 customer questions a month. It wants an assistant that answersfrom the articles, shows which ones it used, and hands the question to a person when it is not sure, all in under 5 seconds and within abudget. Building a demo takes an afternoon. Knowing whether it is good enough to put in front of customers is the work.

2,000 articles, 30,000 questions a month: answer with citations or hand off, in under 5 s.

The estimate

Cost follows tokens, and most tokens are the articles

Tokens for one question, with retrievalSystem prompt400Question60Conversation so far2005 article chunks2,000Answer (output)2502,660 in, 250 outA month, 30,000 questionsSmall model0.10¢ an answer, $29 a monthLarge model1.17¢ an answer, $352 a monthexample prices: see the chapterCost follows tokens, and with retrieval most of the tokens are the articles, not the question.A question is about 60 tokens, but the prompt around it, with 5 chunks of articles, is 2,660. At example prices a small model costs0.10¢ an answer, about $29 a month; a large one 1.17¢, about $352. Both are small beside the cost of a personanswering each question, so the budget question is less "can we afford it" than "is each extra step worth its cost and delay".

2,660 tokens in, 250 out. About $29 a month on a small model, $352 on a large one.

The first design

Asked directly, a model answers from training, not from your product

Customer questionLLM“You are a helpful supportagent for our product.”AnswerQ: Can I export reports to Excel?A: Yes. Open Reports, choose Export, then XLSX. This is free on every plan.Fluent and confident. In fact export is CSV only, and only on the Business plan.Asked directly, a model answers from what it learned in training, which knows nothing about this product.The first version sends the question straight to a model with a short system prompt: 460 tokens in, cheap and fast. The demo impresseseveryone, because every answer reads well. But the model has never seen this company’s help centre. It answers from what similarproducts usually do, so it describes features that do not exist and prices that changed last year, in the same confident tone.

The question goes straight to the model. It answers fluently, and invents a feature and a price.

The cause

Measured on 200 real questions, 42% of answers are wrong

200 real past questions, each scored against the right answer by a judge model and checked by handCorrect116 of 200, 58%Made up42 of 200, 21%Out of date24 of 200, 12%Wrong product14 of 200, 7%Unsafe4 of 200, 2%Measured on 200 real questions, 42% of the answers are wrong, and most of them sound right.An evaluation set turns "it seems good" into a number. 200 past questions with answers written by support staff are put to the assistant,and each answer is graded. 58% are correct. The rest fall into groups: 21% made up, 12% out of date, 7% about the wrong product, and 4 thatwere unsafe, such as advice on bypassing a security setting. Each group points to a different fix.

An evaluation set of 200 past questions: 58% correct; the rest made up, out of date, wrong product or unsafe.

The fix

Retrieval gives the facts; a guardrail and a hand-off stop the rest

Searchkeywords + vectorsRerankkeep the best 5Answeronly from sources, citeGuardrailgrounded? safe?Send or hand offnothing found → personBeforeAftercorrect 172, handed to a person 18, wrong 10 (was 84)Retrieval gives the model the facts; a guardrail and a hand-off stop the rest. Wrong answers fall from 84 to 10.The articles are split into chunks and indexed for keyword and vector search. For each question the best matches are reranked, and the modelis told to answer only from them and cite each one. A guardrail checks the answer is grounded in those chunks and safe before it is sent.When search finds nothing relevant, the question goes to a person. On the same 200 questions: 172 correct, 18 handed off, 10 wrong.

Search, rerank, answer only from sources, check, or hand off. Wrong answers fall from 84 to 10 of 200.

The trade-off

Every step that improves answers adds cost and delay

OptionCost an answerA monthp95 timeCorrectSmall model + retrieval0.10¢$292.4 s81%Small + reranker + guardrail0.28¢$833.9 s86%chosenLarge + reranker + guardrail1.35¢$4056.2 s88%over 5 sEvery step that makes answers better adds cost and delay. Choose against the requirements, not the leaderboard.A reranker and a guardrail on the small model add 5 points of correct answers for 0.18¢ and 1.5 s more per answer. A large model adds2 more points but costs about 5 times as much and breaks the 5-second limit. Good retrieval did more for quality than a bigger model. Theguardrail runs the large model only on the 15% of answers a cheap similarity check flags, which keeps its cost down.

The small model with a reranker and guardrail: 0.28¢ an answer, 3.9 s. A large model breaks the 5 s limit.

The finished design

The model is one box of eleven

Index, nightlyHelp articleschanged ones only · L4Chunkby heading, ~400 tokens · L10Embedvectors · L8Indexkeywords + vectors · L10Each questionSearch, reranktop 5 · L10Promptsources, JSON, cites · L9Small LLManswer · L8Guardrailgrounded, safe · L13Send or hand offa tool call · L11Evaluation: 200 questions, every change (L10)Monitoring: cost, latency, drift (L7)The model is one box of eleven. Most of the quality comes from the boxes around it.Two pipelines: one keeps the index in step with the help centre each night, the other answers each question. Around them, the evaluationset runs on every change to a prompt, a model or the chunking, and blocks a change that scores worse. Monitoring watches cost, latencyand the hand-off rate. Each box opens the lesson that teaches it (L is the lesson number).

An index pipeline, a query pipeline, a guardrail, evaluation and monitoring. Each box links to its lesson.

A real case

At DoorDash, the guardrail and the judge did most of the work

RAG for delivery driversfinds similar solved cases,puts their help articlesinto the promptTwo-tier guardrailcheap similarity check first;an LLM reviews what it flagsfor grounding and complianceLLM judgescores retrieval, accuracy,grammar, coherence andrelevance over timeReportedhallucinations down 90%; potentially severe compliance issues down 99%Andprompt tests run on every change; a person takes over when neededAt DoorDash, the guardrail and the judge, not the model, did most of the work of making answers safe.DoorDash built a RAG assistant for its delivery drivers. Every answer must pass a guardrail before a driver sees it: a cheap similaritycheck against the articles, then an LLM review of anything it flags. An LLM judge scores quality over time, and tests run on every promptchange. DoorDash reports 90% fewer hallucinations and 99% fewer potentially severe compliance issues. The same shape as our design.

DoorDash’s driver assistant: RAG, a two-tier guardrail and an LLM judge. Hallucinations down 90%.

Lesson complete

You finished “A support assistant over company documents”

Nicely done.

Open in a new tab