RAG (retrieval-augmented generation)
RAG lets a model answer from your own documents by fetching the relevant passages first and placing them into the prompt — knowledge without retraining anything.
The recipe has three steps: your documents are split into passages and indexed by meaning; when you ask a question, the passages most related to it are retrieved; those passages go into the context window alongside your question, and the model answers with them in view. Nothing about the model changes — it simply reads before it speaks.
That is why local people care about it: a private manual, a codebase or ten years of notes can stay on your machine and still be searchable by a model that has never seen them. It is also the sensible alternative to fine-tuning for anything factual — facts belong in retrievable text, style belongs in weights.
Two costs are easy to miss. Retrieved passages are tokens, so they eat context and KV-cache memory on every question. And the answer is only as good as the retrieval — feed the model the wrong passage and it will explain the wrong passage fluently.
Where you see it on the radar
RAG lives in the context window: every retrieved passage is tokens the KV cache must hold. The fit engine prices that per context length — which is why the same model can fit at 8K and fail at 32K.
Check fits at longer contexts →Related terms