ByHeartAI
Advanced8 min read

How to Evaluate RAG

Evaluating RAG means measuring its two halves separately — did retrieval fetch the right context, and did the model faithfully use it — so you know exactly what to fix.

Explain like I'm new to AI

"The answers seem bad" isn't actionable. To improve RAG you must know which part failed: did the system retrieve the wrong information, or did it retrieve the right information but answer badly?

So RAG evaluation splits into two halves — retrieval quality and generation quality:

Evaluate both halves: if retrieval fails, even a perfect model answers wrong — measure each separately.

Mental model

An open-book exam has two ways to fail: you flipped to the wrong page (retrieval), or you read the right page but wrote a wrong answer (generation). Grading both separately tells you whether to study your search or your writing.

How it works — the core metrics

Retrieval quality:

  • Context precision: of the chunks retrieved, how many are actually relevant? (Low = too much noise.)
  • Context recall: of the chunks needed to answer, how many did we retrieve? (Low = missing information.)

Generation quality:

  • Faithfulness / groundedness: is every claim in the answer supported by the retrieved context? (Low = hallucination.)
  • Answer relevance: does the answer actually address the question?
  • Answer correctness: does it match the known ground-truth answer (when you have one)?

Real-world example

Your bot gives a wrong refund policy. Evaluation shows context recall is low — the right chunk was never retrieved. Now you know to fix retrieval (chunking, hybrid search, reranking), not the prompt. In another case, faithfulness is low while context recall is high: the right chunk was retrieved, but the model ignored it — fix the prompt/grounding instead. Same symptom, opposite fixes — only measurement tells them apart.

Technical explanation

Build a test set of representative questions with (ideally) reference answers and/or the "gold" chunks. Then:

  • Score retrieval with precision/recall (needs labeled relevant chunks).
  • Score generation with LLM-as-a-judge for faithfulness/relevance (a strong model grades each answer against the context), plus human review on a sample.
  • Track metrics over time and in CI so changes (new embeddings, chunking, prompts) can't silently regress quality.

Open frameworks (e.g. RAGAS-style toolkits) automate many of these metrics. Golden rule: evaluate retrieval and generation independently — averaging them hides which stage is broken.

Common mistakes

Common mistake

Only looking at the final answer. If you don't measure retrieval separately, you can't tell whether a bad answer came from bad context or bad generation — and you'll "fix" the wrong thing.

  • Testing on a handful of cherry-picked questions instead of a representative set.
  • No regression testing, so an "improvement" quietly degrades another metric.

When to use it

  • Before and continuously after shipping any RAG system — evaluation is how you improve it safely.

When NOT to use it

  • Never skip it for production; only throwaway prototypes can defer formal evaluation.

Alternatives

  • There's no substitute for measurement; the choice is only which metrics and how much automation vs. human review.

Quick quiz

Question 1 of 3

RAG evaluation splits into which two halves?

Question 2 of 3

What does 'faithfulness' measure?

Question 3 of 3

A bad answer has high context recall but low faithfulness. What should you fix?

Related concepts

  • RAG Architecture — RAG has two phases — offline ingestion (chunk, embed, store) and per-query answering (retrieve, rerank, generate) — connected by a vector store.
  • Agentic RAG & Advanced Patterns — Agentic RAG turns the one-shot pipeline into a self-correcting loop that judges its own retrieval and answers, plus patterns like Graph RAG and multi-hop.
  • Why Evaluation Matters — Without a frozen eval set, every prompt, RAG, or model change is a guess — evaluation is how you know the system actually got better.
  • LLM-as-a-Judge — LLM-as-a-judge grades outputs against a rubric at scale. Calibrate it on humans, and watch verbosity, position, and self-preference bias.
  • Tool, Retrieval, and Agent Traces — Nested spans for retrieve, each model call, and each tool — a waterfall that shows which stage failed, without dumping secret payloads.
  • RAG Poisoning — RAG poisoning plants text in the index so retrieval serves it as context. Control who can write; treat hits as untrusted even when they look grounded.
NextRAG vs Fine-tuning vs Long Context

Last reviewed: 2026-09-01 · Written by ByHeart AI · Reviewed by ByHeart AI