How to Evaluate a RAG System
Measure RAG quality with retrieval and generation metrics, a golden set, and the right eval tools so you can find which stage fails.
Evaluating a RAG system means measuring two stages separately: retrieval quality (did the right chunks come back?) and generation quality (did the model use them faithfully?). Score retrieval with recall@k, precision@k, MRR, and nDCG; score generation with faithfulness, answer relevancy, context precision, and context recall. The whole point is diagnosing which stage failed, because a faithful answer built on the wrong context is still a wrong answer.
If you only track one number, you cannot fix anything. A single end-to-end score tells you the system is bad but not why. This guide breaks evaluation into the parts you can actually act on, then maps them to the 2026 tooling so you know what to install.
What “good” means for a RAG system
A retrieval-augmented generation pipeline has two moving parts that fail independently. Retrieval pulls candidate chunks from your index. Generation reads those chunks and writes an answer. When the final answer is wrong, the failure lives in one of those two stages, and you need to know which.
Think of it as a two-stage failure model:
- Retrieval failed: the right facts never reached the model. No amount of prompt tuning fixes this.
- Generation failed: the right facts were in context, but the model ignored them, misread them, or invented something. This is a hallucination problem.
A faithful answer over the wrong context still ships a wrong answer to your user. So you measure both stages, and you keep the scores apart. Collapsing them into one metric is the most common evaluation mistake.
Retrieval metrics: recall@k, precision@k, MRR, nDCG, hit@k
These metrics ask one question: given a query, did the retriever return the documents that actually answer it, and did it rank them well? All of them depend on knowing which documents are relevant, which is why a golden set (next section) comes first.
| Metric | What it measures | When to choose |
|---|---|---|
| recall@k | Share of all relevant docs that appear in the top k | Your top concern is missing facts; low recall starves the model |
| precision@k | Share of the top k that is actually relevant | Your context window fills with noise the model must wade through |
| hit@k | Whether any relevant doc appears in the top k | You need a coarse pass/fail signal per query |
| MRR | Rewards the first relevant doc being ranked early | One correct chunk is enough and position matters |
| nDCG | Graded relevance with position weighting | Some docs are more relevant than others, not just yes/no |
Two failure modes drive most retrieval problems. Low recall@k means relevant documents never made it into the top k, so the model has no facts to work with and drifts toward incomplete answers or hallucination. Low precision@k means the top k is padded with irrelevant chunks, so the model has to find the signal in the noise, which also raises hallucination risk. Both are retrieval failures, not generation failures, and both push the same symptom (a bad answer) for opposite reasons.
MRR and nDCG matter more than people expect. Many models weight earlier context more heavily than later context, so ranking the relevant chunk first improves the answer even when the same chunk would eventually appear lower in the list. Getting the order right is not cosmetic.
Generation metrics: faithfulness, answer relevancy, context precision, context recall
Once you trust that retrieval works, you measure what the model did with the context. The de facto standard here is RAGAS, the most widely adopted open-source RAG eval framework in 2026. It is lightweight, largely reference-free, and built around four core metrics. Most of them are LLM-as-judge: an LLM acts as a structured grader and its judgments become numeric scores from 0 to 1, rather than string matching against a reference.
- Faithfulness: decomposes the generated answer into atomic claims and verifies each claim against the retrieved context. The score is the proportion of claims supported by context, 0 to 1. This is your primary hallucination metric.
- Answer relevancy: how well the answer addresses the actual question, penalizing padding and off-topic content.
- Context precision: computed as mean precision@k across retrieved chunks, so it captures how well relevant chunks are ranked.
- Context recall: the ratio of relevant claims found in the retrieved results to the total relevant claims in the reference answer. It tells you whether anything important was missed.
The composite “ragas score” is the mean of faithfulness, answer relevancy, context recall, and context precision. Treat the composite as a dashboard headline, never as a diagnosis. When it drops, you look at the four components to find the culprit. Context recall in particular needs reference ground truth to compute, which is the reason the golden set is non-negotiable.
Build a golden set before you measure anything
A golden set is a curated collection of question plus reference-answer plus expected-context examples. It is the prerequisite for measuring almost everything above. Without reference answers you cannot compute context recall, and you cannot tell whether a faithful answer is also correct.
Build it like this:
- Collect real questions your users ask, not synthetic ones you invented.
- Write the correct reference answer for each.
- Tag the specific documents or chunks that should be retrieved to answer it.
- Keep it small and high quality to start. Fifty good examples beat five hundred noisy ones.
The golden set is not a one-time artifact. You grow it continuously by annotating production failures: when a user gets a bad answer, a human reviews it, writes the correct answer, and adds it to the set. That loop is what keeps evaluation honest as your content and your users change.
Why your retrieval is probably the bottleneck (and how to prove it)
Teams reach for prompt engineering when answers are bad, but retrieval is usually where the loss happens. Here is how to prove it instead of guessing.
Run your golden set and read context recall and faithfulness side by side. If context recall is low, the retriever is failing and prompt changes will not help, because the facts are not in the window. If context recall is high but faithfulness is low, retrieval did its job and the generation step is the problem. This single comparison tells you which half of the system to fix.
When retrieval is the bottleneck, chunking strategy is one of the highest-leverage fixes. Embedding atomic units such as individual sentences instead of whole chunks measurably raises recall, because a whole chunk mixes several semantic pieces while a query usually targets one. Smaller, cleaner embeddings match the intent of a query more precisely. Your vector database returns better candidates when each stored unit carries a single idea.
The 2026 tooling landscape
No single tool does everything well. Each leans into a different part of the lifecycle.
| Tool | Strength | When to choose |
|---|---|---|
| RAGAS | The four core RAG metrics, reference-free, lightweight | You want a fast, framework-agnostic eval baseline |
| DeepEval | Broadest metric library, strongest CI/CD integration beyond RAG | You need eval gating across many task types, not just RAG |
| Arize Phoenix | Projects query and document embeddings into 2D/3D | You need to see retrieval drift visually |
| TruLens | RAG metrics paired with OpenTelemetry span-level tracing | You already run OTel and want eval inside your traces |
| LangSmith | Auto-traces every LangChain/LangGraph step | You build on LangChain and want zero-instrumentation traces |
| Braintrust | The most RAG scorers out of the box, CI/CD gating | You want broad scorers and a managed annotation platform |
The practitioner consensus in 2026 is that you almost certainly need two tools, not one. Pair a lightweight framework for CI/CD gating (DeepEval, RAGAS, or Promptfoo) with a platform for human annotation and regression tracking (Braintrust, LangSmith, or Arize Phoenix). The first gates merges; the second feeds the golden set and tracks regressions over time.
A practical evaluation workflow
The eval lifecycle runs in three phases, and each uses different tools for different reasons.
- Development evals: run your golden set locally while you change chunking, retrievers, or prompts. Fast feedback, run on every code change.
- CI/CD gating: wire a lightweight framework into the pipeline so a pull request that drops faithfulness below a threshold fails the build. This stops regressions before they ship.
- Production monitoring: trace live traffic, sample real answers, and route failures to human annotation. Those annotations flow back into the golden set, closing the loop.
The flow is development to automated gating to production monitoring, with human annotation continuously feeding the golden set. Skipping the gating step is how silent regressions reach users.
Context trustworthiness: the eval dimension the metrics assume away
The four standard metrics share a blind spot. They treat the index as trustworthy by default. Faithfulness asks whether the answer matches the retrieved context, not whether that context is correct. So a stale, outdated, or wrong source scores as perfectly “faithful” while being factually wrong.
2026 sources flag this as a fifth dimension: context trustworthiness, meaning the ownership, freshness, and lineage integrity of the index itself. You cannot measure it with an LLM judge over a single answer. You manage it with process: track who owns each source, when it was last updated, and where it came from. An answer faithfully grounded in a document that went out of date six months ago is a trap the metrics will not catch for you.
From eval to fix: turning bad scores into a better pipeline
Scores are only useful if they point at a fix. Map each failing metric to its lever:
- Low recall@k or low context recall: fix retrieval. Re-chunk to smaller atomic units, improve the embedding model, or expand k.
- Low precision@k: tighten retrieval. Add reranking, raise the similarity threshold, or filter by metadata.
- Low MRR or nDCG: add a reranker so the best chunk lands first.
- Low faithfulness: constrain generation. Tighten the prompt to forbid outside knowledge, or switch to a model that follows context better.
- Low answer relevancy: the model is padding or wandering. Trim the prompt and ask for direct answers.
- High faithfulness but wrong facts: this is a context trustworthiness problem. Audit the source, not the model.
Evaluation is not a report you file once. It is the instrument panel you keep watching while you change the pipeline. If you are building this on internal data, our private RAG solution covers the same retrieval, generation, and trust dimensions end to end. For the broader picture of how the pieces connect, start from the RAG glossary entry and work outward.