RAG in Production: Build, Debug, Improve

RAG stands for Retrieval Augmented Generation. Your code first searches your own documents for passages that relate to the question, then gives only those passages to a language model and asks it to answer from them. It works like an open-book exam: the model does not need to remember the facts because it can read them.

Yes, for most real systems. Sending hundreds of pages with every question is slow and expensive, and every user would see every document. Retrieval also lets you filter by permission and cite the exact source. Pasting the whole text is fine for one short document.

There is no universal size. Start by splitting along the structure of the document, such as sections and paragraphs, and keep each chunk to a few hundred tokens. Then measure on real questions and adjust, because the right size depends on how your documents are written.

Most of the time retrieval failed, not the model. The correct passage was never found, was ranked too low, or was filtered out. Check each stage in order: is the fact in the index, in the candidate list, in the final top results, and in the prompt?

Use RAG when answers must come from documents that change or must be cited. Use fine-tuning to change behaviour, such as tone or output format. Many teams try better prompting first, then RAG, and fine-tune only when the first two are not enough.