RAG: give your language model a library card

Retrieval-augmented generation combines finding relevant material with generating a response. Instead of relying only on information represented in model parameters, a system retrieves external passages and supplies them as context. Think library card, not truth serum.

The Lewis and colleagues RAG paper combined a pretrained generator with a dense retrieval index and studied knowledge-intensive tasks. The Atlas research later examined retrieval-augmented models in few-shot settings. These are foundations; present-day products use many architectural variations.

The pipeline has several places to trip

A practical document assistant ingests material, extracts text, divides it into passages, indexes them, retrieves candidates, optionally reranks them, and generates an answer. Source identifiers connect the output to the underlying material. Each stage can introduce a failure.

Bad extraction loses a table header. Bad chunking separates an exception from the rule. Bad retrieval finds an outdated policy. Bad generation invents a connection. A beautiful citation can point to a passage that does not actually support the sentence. A reference is a route to evidence, not an automatic endorsement.

Start with source quality

Keep document dates, owners, versions, and access rules. Prefer authoritative material over duplicates and stale drafts. If a document is withdrawn, remove it from retrieval promptly. If a user lacks permission to read a document, filter before returning its content to the model.

A toy corpus can be public. An organizational corpus often cannot. Retrieval authorization is part of the security model, not an optional polish pass after search looks good.

Choose chunks for the question

The right chunk size depends on the material and task. A short definition and a multi-page contractual condition need different context. Keep headings and source metadata with passages. Use overlap when it helps retain meaning, but do not assume more overlap always improves retrieval.

Hybrid retrieval can combine exact lexical matches with vector similarity. A product identifier may need an exact match; a paraphrased customer question may benefit from semantics. Dense retrieval research such as DPR shows the value of learned representations, but there is no rule requiring every search problem to abandon lexical methods.

Evaluate retrieval separately

Create questions with known supporting passages. Measure whether the correct evidence appears among retrieved candidates before judging the answer. If the source never enters context, a better prompt cannot reliably conjure it.

Then test answer support. Does each important statement follow from the cited material? Does the system distinguish old from current versions? Does it say it cannot answer when the corpus lacks evidence? Include conflicting documents and deliberately irrelevant passages.

Keep instructions out of the evidence lane

A retrieved document might say “ignore all previous instructions.” That is document content, not an authorized command. Restrict tools and treat external text as untrusted input. A model prompt is one layer; permission design and action validation are others.

An example worth testing

Ask, “Can contractors claim meal expenses?” A correct answer might need the contractor policy, a spending cap, and a regional exception. Test those cases explicitly. An answer copied from the employee handbook can sound polished and be entirely wrong for the user.

RAG shines when current, inspectable source material matters. It adds a path to grounded answers; it also adds an index that someone must maintain. Libraries work better with librarians. Read embeddings and the evaluation guide before issuing your chatbot a membership card.