Production RAG: Retrieval Quality, Evaluation, and Guardrails

Retrieval-augmented generation (RAG) is often described as “search, then paste results into a prompt.” That is a useful prototype, but it hides the hard parts: document quality, chunk boundaries, hybrid retrieval, reranking, citations, evaluation, and access control.

RAG is a data system

The language model is the visible part, but retrieval quality sets the ceiling. If the relevant evidence never reaches the context window, prompt engineering cannot recover it.

Treat the knowledge pipeline as a first-class data product:

source → parse → normalize → enrich → chunk → index → retrieve → rerank → cite

Every stage needs ownership, observability, and a way to be replayed.

Ingestion: preserve meaning and provenance

Documents are not plain strings. A PDF has headings, tables, footnotes, and page numbers. A support article has a product version and effective date. A database row has tenant permissions.

Store metadata alongside every chunk:

• source ID and canonical URL;
• document version and updated timestamp;
• section hierarchy;
• page or paragraph location;
• tenant and access-control attributes;
• content hash for deduplication.

Without provenance, you cannot produce trustworthy citations or remove obsolete content reliably.

Chunk by semantic structure

Fixed-size chunks are a baseline, not a strategy. They often split a definition from its conditions or a code sample from its explanation. Prefer structure-aware chunking: headings, paragraphs, list groups, table rows, and function boundaries.

Use overlap only when it protects meaning. Excessive overlap inflates the index and causes near-duplicate results that crowd useful evidence out of the context window.

A good chunk is independently understandable, narrow enough to retrieve precisely, and rich enough to answer a real question.

Combine lexical and semantic retrieval

Embeddings are strong at meaning; lexical search is strong at exact identifiers, error codes, names, and product versions. Production systems usually need both.

candidate_set = union(
    vector_search(query_embedding),
    keyword_search(query_text),
    metadata_filters(user_scope)
)

Fuse the scores, then rerank the best candidates with a model that evaluates query-document relevance. Retrieval gets broad recall; reranking restores precision.

Apply permissions before generation

Authorization must be part of retrieval, not a disclaimer in the prompt. Filter by tenant, role, document visibility, and lifecycle status before a chunk can enter the candidate set.

The safe sequence is:

1. resolve the authenticated principal;
2. derive allowed scopes;
3. apply metadata filters during retrieval;
4. log the IDs of retrieved chunks;
5. generate only from authorized evidence.

Post-filtering after retrieval can leak sensitive text into traces, caches, or model requests.

Make grounding explicit

Ask the model to distinguish evidence from inference. Require citations at the claim level and provide a clear abstention behavior when evidence is missing or contradictory.

{
  "answer": "...",
  "citations": [
    {"source_id": "policy-42", "section": "Refund window"}
  ],
  "grounded": true,
  "missing_information": []
}

Then validate that every cited source was actually included in the retrieved context. A well-formatted citation that points nowhere is worse than no citation.

Evaluate retrieval and generation separately

End-to-end answer quality alone does not tell you where the failure occurred. Split evaluation into layers.

Retrieval metrics

• Recall@K: did the evidence appear in the candidate set?
• Precision@K: how much retrieved content was truly relevant?
• Mean reciprocal rank: how early did the best evidence appear?
• Filter correctness: did access rules include and exclude the right documents?

Generation metrics

• faithfulness to retrieved evidence;
• citation correctness and completeness;
• answer relevance;
• appropriate abstention;
• format and policy compliance.

Build the evaluation set from real user questions. Synthetic questions help bootstrap coverage, but production queries reveal vocabulary, ambiguity, and missing knowledge that internal teams rarely predict.

Handle freshness deliberately

RAG systems silently degrade when indexes become stale. Use content hashes, incremental indexing, deletion propagation, and freshness service-level objectives. Record the index version in every trace so an answer can be reproduced.

For volatile facts, route to authoritative APIs rather than embedding snapshots. Retrieval is not limited to vector databases; the best source may be SQL, a search engine, or a live service call.

Observe the full pipeline

For each request, capture:

• normalized query and any query rewrite;
• filters and retrieval strategy;
• candidate IDs and scores;
• reranked order;
• context actually sent to the model;
• answer, citations, latency, and cost;
• user feedback or downstream outcome.

Redact sensitive data, but keep enough structure to diagnose failures. “The model hallucinated” is not a diagnosis. The evidence may have been missing, badly ranked, outdated, or ambiguous.

Production checklist

Before calling a RAG system ready, verify:

• structure-aware ingestion with provenance;
• hybrid retrieval and reranking;
• permission filters at query time;
• citations validated against retrieved chunks;
• retrieval and generation evaluated independently;
• index freshness and deletion handling;
• traces that make failures reproducible;
• an explicit “I do not have enough evidence” path.

Good RAG is not more context. It is the smallest set of authorized, current, relevant evidence that supports a defensible answer.