Your RAG pipeline demos beautifully. You ask it a question about the product docs, it retrieves the right chunk, and the answer is crisp and accurate. You ship it. Three weeks later, a user asks a slightly rephrased question, gets a hallucinated answer with 90% confidence, and files a bug report that says "your AI is lying."
You can't debug what you can't measure. And most RAG pipelines ship with zero quantitative evaluation—no test dataset, no metrics, no regression suite. Just vibes and a handful of manual spot-checks.
Ragas solves this. It gives you LLM-based metrics that score retrieval quality, generation faithfulness, and answer correctness against a structured test dataset. But the tool is only as useful as the dataset you feed it and your ability to interpret what the scores actually mean.
This article walks through building a real evaluation pipeline with Ragas—from generating a test dataset to running metrics to diagnosing the specific failure mode when your faithfulness score drops from 0.92 to 0.71 after a chunking change.
TL;DR
- Ragas evaluates RAG pipelines using LLM-as-judge metrics (faithfulness, relevancy, context precision/recall)
- You need a structured test dataset: questions, retrieved contexts, generated answers, and ground-truth references
- Synthetic test generation gets you started fast; manual curation catches edge cases synthetic misses
- Run evaluation in CI. A 5% drop in context recall after a retriever change is a regression, not noise
- Low scores are diagnostic: they tell you which component (retriever vs. generator) is failing
📋 Table of Contents
- The Mental Model: What Ragas Actually Evaluates
- Generating a Synthetic Test Dataset from Your Documents
- Curating a Manual Test Set for the Cases That Matter
- The Four Metrics That Tell You Where Your Pipeline Breaks
- Running Evaluation and Reading the Results
- Diagnosing Failures: What a Score Drop Actually Means
- Wiring Ragas into CI for Regression Detection
- Seven Mistakes That Make Your Evaluation Useless
- When Ragas Isn't the Right Tool
1. The Mental Model: What Ragas Actually Evaluates
A RAG pipeline has two distinct failure surfaces:
Retrieval failure: The right information exists in your corpus but the retriever didn't find it, or ranked it too low, or returned irrelevant chunks that pollute the context.
Generation failure: The retriever found the right chunks, but the LLM hallucinated, ignored the context, or produced an answer that doesn't address the question.
Traditional testing treats the pipeline as a black box: input question → check output. Ragas decomposes it. It scores the retrieval step and the generation step independently, so you know whether to fix your embedding model and chunking strategy or your prompt and model choice.
The evaluation loop looks like this:
Test Dataset → Your RAG Pipeline → Evaluation Metrics → Scores per sample
↓ ↓
(question, ground truth) (faithfulness, relevancy,
context precision, recall)
Each test sample needs four fields:
| Field | What it is | Who provides it |
|---|---|---|
user_input | The question | Test dataset |
retrieved_contexts | Chunks the retriever returned | Your pipeline at eval time |
response | The LLM's generated answer | Your pipeline at eval time |
reference | The ground-truth correct answer | Test dataset (human or synthetic) |
The reference field is what separates evaluation from just "running the pipeline." Without it, you can't measure recall or correctness. You can only measure internal consistency.
2. Generating a Synthetic Test Dataset from Your Documents
Scenario: You have 200 pages of product documentation and zero labeled Q&A pairs. You need a test set by Friday.
Ragas includes a TestsetGenerator that builds a knowledge graph from your documents and generates question-answer pairs with varying complexity (single-hop facts, multi-hop reasoning, abstract summarization).
from ragas.testset import TestsetGenerator
from ragas.llms import LangchainLLMWrapper
from ragas.embeddings import LangchainEmbeddingsWrapper
from langchain_openai import ChatOpenAI, OpenAIEmbeddings
from langchain_community.document_loaders import DirectoryLoader, TextLoader
# Load your document corpus
loader = DirectoryLoader("./docs/", glob="**/*.md", loader_cls=TextLoader)
documents = loader.load()
# Configure the LLM and embeddings used for generation
generator_llm = LangchainLLMWrapper(ChatOpenAI(model="gpt-4o", temperature=0.3))
generator_embeddings = LangchainEmbeddingsWrapper(
OpenAIEmbeddings(model="text-embedding-3-small")
)
# Build the test set
generator = TestsetGenerator(
llm=generator_llm,
embedding_model=generator_embeddings,
)
testset = generator.generate_with_langchain_docs(
documents,
testset_size=50, # Number of Q&A pairs to generate
)
# Convert to a pandas DataFrame for inspection
df = testset.to_pandas()
print(df[["user_input", "reference", "reference_contexts"]].head())
The generator produces three question types by default:
- Single-hop: Answer lives in one chunk. ("What's the default timeout for API requests?")
- Multi-hop: Requires combining info from two or more chunks. ("How does the retry policy interact with the rate limiter?")
- Abstract/synthetic: Requires reasoning across the document. ("Summarize the security model and its limitations.")
💡 Practical note: Generate more samples than you need—say 80—and manually filter to 50. The generator occasionally produces questions that are trivially answerable or ambiguously worded. You want a test set that challenges your pipeline, not one that inflates your scores.
Save your test set. Serialize it to disk so you reuse the exact same questions across runs:
testset.to_pandas().to_parquet("eval/testset_v1.parquet")
Versioning your test dataset matters as much as versioning your code. If you regenerate questions every run, you can't compare scores across deployments.
3. Curating a Manual Test Set for the Cases That Matter
Synthetic generation covers breadth. It won't cover the specific failure modes your users actually hit.
The questions that break RAG pipelines in production are usually:
- Negations: "What features are NOT included in the free tier?"
- Comparative: "How does plan A differ from plan B regarding data retention?"
- Temporal: "What changed in the API between v2 and v3?"
- Adversarial rephrasing: "Does this thing work if I do the opposite of what the docs say?"
Build a small manual set (20–40 samples) targeting these patterns. The format matches what Ragas expects:
from ragas.dataset_schema import EvaluationDataset, SingleTurnSample
manual_samples = [
SingleTurnSample(
user_input="What happens if I exceed the rate limit on the Pro plan?",
reference="On the Pro plan, exceeding the rate limit returns a 429 status code. "
"Requests are queued for up to 30 seconds before failing. "
"No data is lost; clients should implement exponential backoff.",
),
SingleTurnSample(
user_input="Which authentication methods are NOT supported for service accounts?",
reference="Service accounts do not support OAuth2 interactive flow or SAML. "
"Only API key and JWT bearer token authentication are available.",
),
# ... more samples
]
manual_dataset = EvaluationDataset(samples=manual_samples)
Notice there's no retrieved_contexts or response field here. Those get filled in when you run your pipeline against the dataset. You provide the question and the ground truth; Ragas fills in the pipeline outputs during evaluation.
⚠️ Gotcha: Your
referenceanswers must be grounded in your actual documents. If you write a reference answer from memory that contradicts what's in the corpus, your context recall metric will be misleadingly low. The retrieval step can't find information that isn't there.
4. The Four Metrics That Tell You Where Your Pipeline Breaks
Ragas ships with many metrics, but four carry most of the diagnostic weight:
| Metric | What it measures | Failure it detects | Component |
|---|---|---|---|
| Faithfulness | Is the answer grounded in retrieved context? | Hallucination | Generator |
| Answer Relevancy | Does the answer address the question? | Evasion, off-topic generation | Generator |
| Context Precision | Are relevant chunks ranked above irrelevant ones? | Poor retrieval ranking | Retriever |
| Context Recall | Did retrieval find all info needed for the reference answer? | Missing retrieval | Retriever |
Two additional metrics worth knowing:
- Answer Correctness (semantic similarity + factual overlap between
responseandreference): Overall end-to-end quality. - Noise Sensitivity: Does the answer degrade when irrelevant chunks are injected into context?
The critical insight: faithfulness and context recall are independent. You can have perfect faithfulness (the LLM only says what's in the context) but terrible recall (the retriever missed half the relevant information). The answer is "faithful" but incomplete. Conversely, high recall with low faithfulness means the retriever found the right chunks but the LLM invented details not present in them.
from ragas.metrics import (
Faithfulness,
AnswerRelevancy,
ContextPrecision,
ContextRecall,
)
Each metric uses an LLM (typically GPT-4o or Claude Sonnet) as a judge. The judge LLM decomposes the answer into claims, checks each claim against the context, and produces a 0–1 score. This is why running Ragas evaluation costs real API tokens—budget accordingly.
5. Running Evaluation and Reading the Results
With your dataset prepared and your pipeline producing retrieved_contexts and response for each sample, evaluation is a single call:
from ragas import evaluate
from ragas.llms import LangchainLLMWrapper
from langchain_openai import ChatOpenAI
# The LLM used as judge (separate from your pipeline's LLM)
judge_llm = LangchainLLMWrapper(ChatOpenAI(model="gpt-4o", temperature=0))
# Run your pipeline to fill in retrieved_contexts and response for each sample
# (This step depends on your RAG implementation—LangChain, LlamaIndex, custom, etc.)
populated_dataset = run_pipeline_on_dataset(manual_dataset) # Your code here
results = evaluate(
dataset=populated_dataset,
metrics=[
Faithfulness(),
AnswerRelevancy(),
ContextPrecision(),
ContextRecall(),
],
llm=judge_llm,
)
# Aggregate scores
print(results)
# {'faithfulness': 0.87, 'answer_relevancy': 0.91,
# 'context_precision': 0.78, 'context_recall': 0.72}
# Per-sample breakdown for debugging
df = results.to_pandas()
low_recall = df[df["context_recall"] < 0.6]
print(low_recall[["user_input", "context_recall"]])
The aggregate scores give you a health check. The per-sample DataFrame gives you a debugging tool. Sort by the lowest-scoring metric, read those specific questions, and you'll find patterns: "All multi-hop questions have recall < 0.5" tells you your chunk size is too small to capture cross-document relationships.
What "good" scores look like (rough guidance, not absolute):
| Metric | Concerning | Acceptable | Strong |
|---|---|---|---|
| Faithfulness | < 0.7 | 0.7–0.85 | > 0.85 |
| Answer Relevancy | < 0.75 | 0.75–0.88 | > 0.88 |
| Context Precision | < 0.6 | 0.6–0.8 | > 0.8 |
| Context Recall | < 0.65 | 0.65–0.8 | > 0.8 |
These ranges shift depending on domain difficulty. Legal document Q&A will naturally score lower on recall than FAQ-style support because the information is denser and more scattered.
6. Diagnosing Failures: What a Score Drop Actually Means
Scores without diagnosis are just anxiety. Here's the interpretive framework:
Faithfulness dropped but recall is fine:
Your retriever works. Your generator is hallucinating. Check: Is the system prompt encouraging the model to "fill in gaps"? Is the temperature too high? Is the context window so long that the model loses track of what's actually in the retrieved chunks?
Context recall dropped but faithfulness is fine:
The generator is honest but under-informed. The retriever is missing chunks. Check: Did you change the chunk size? Did you switch embedding models? Is the query being rewritten in a way that loses keywords? Try increasing top_k or adjusting your overlap between chunks.
Context precision dropped:
The retriever returns relevant chunks but buries them below irrelevant ones. Your ranking is off. Check: Are you using a reranker? Is your embedding model appropriate for the domain? Are metadata filters working correctly?
Answer relevancy dropped but faithfulness is high:
The model is answering a different question than what was asked. It's faithful to the context but the context doesn't match the intent. This often means query understanding is failing—the retriever fetches chunks for a related-but-wrong topic.
Scenario in practice: You switch from 512-token chunks to 1024-token chunks. Context recall jumps from 0.68 to 0.81 (more information per chunk). But faithfulness drops from 0.89 to 0.82 (the model gets lost in longer contexts and starts paraphrasing loosely). The fix isn't reverting chunk size—it's adding a "cite your sources" instruction to the generation prompt or switching to a model that handles longer contexts more faithfully.
7. Wiring Ragas into CI for Regression Detection
Running evaluation manually after every change doesn't scale. The real value is catching regressions automatically.
# tests/test_rag_quality.py
import pytest
from ragas import evaluate
from ragas.metrics import Faithfulness, ContextRecall
from ragas.dataset_schema import EvaluationDataset
THRESHOLDS = {
"faithfulness": 0.80,
"context_recall": 0.70,
}
@pytest.fixture(scope="module")
def eval_results():
dataset = load_versioned_testset("eval/testset_v1.parquet")
populated = run_pipeline(dataset)
return evaluate(
dataset=populated,
metrics=[Faithfulness(), ContextRecall()],
llm=judge_llm,
)
def test_faithfulness_above_threshold(eval_results):
score = eval_results["faithfulness"]
assert score >= THRESHOLDS["faithfulness"], (
f"Faithfulness regression: {score:.3f} < {THRESHOLDS['faithfulness']}"
)
def test_context_recall_above_threshold(eval_results):
score = eval_results["context_recall"]
assert score >= THRESHOLDS["context_recall"], (
f"Context recall regression: {score:.3f} < {THRESHOLDS['context_recall']}"
)
🚨 Production warning: LLM-as-judge scores have variance. Running the same evaluation twice can produce scores that differ by ±0.03–0.05. Don't set thresholds so tight that normal variance triggers failures. A 0.05 buffer between your expected score and your threshold prevents flaky CI. Alternatively, run each sample 2–3 times and average.
Cost matters here. Evaluating 50 samples with 4 metrics using GPT-4o as judge runs roughly 200 LLM calls per CI run. At current pricing, that's a few dollars per run. Fine for nightly CI. Expensive if you run it on every commit. Gate it: run the full suite nightly, run a 10-sample smoke test on PRs.
8. Seven Mistakes That Make Your Evaluation Useless
-
Evaluating without a reference answer. You can measure faithfulness (is the answer grounded in context?) without ground truth. But you can't measure recall or correctness. You'll miss every case where the retriever fails to find information that should be there.
-
Using your production LLM as the judge. If GPT-4o generates your answers and also judges them, it has a bias toward its own phrasing. Use a different model as judge, or at minimum a different model family.
-
Testing only easy questions. If your test set is all single-hop factoid questions, your scores will look great and your pipeline will still fail on the multi-hop reasoning that users actually need.
-
Not versioning the test set. You change three questions, re-run evaluation, and can't tell if the score change is from your pipeline change or the dataset change.
-
Treating aggregate scores as sufficient. An average faithfulness of 0.85 hides the fact that 15% of samples score 0.4. Look at the distribution, not just the mean.
-
Ignoring cost and latency of evaluation itself. Running Ragas on 500 samples with GPT-4o as judge takes time and money. Start with 50–80 well-chosen samples.
-
Never updating the test set. Your product docs change. Your users ask new questions. A test set frozen six months ago evaluates a pipeline that no longer exists in the same form.
9. When Ragas Isn't the Right Tool
Ragas is built for RAG evaluation specifically. It assumes you have discrete retrieved chunks, a generation step, and a reference answer to compare against. It's not the right fit for:
- Open-ended creative generation where no single "correct" answer exists
- Conversational agents where quality depends on multi-turn coherence rather than single-turn factual accuracy
- Pipelines with no retrieval step (pure LLM generation without RAG)
- Latency or throughput benchmarking (Ragas measures quality, not speed)
For those cases, you'd reach for different tools: custom LLM-as-judge rubrics, human evaluation panels, or task-specific metrics like BLEU/ROUGE for summarization.
There's also a practical ceiling: LLM-as-judge has its own failure modes. The judge model can misparse complex answers, disagree with itself across runs, or fail to evaluate highly technical domain content accurately. For medical, legal, or financial RAG systems, supplement Ragas scores with periodic human review of the lowest-scoring samples.
The Evaluation Loop That Actually Ships
Here's the minimal viable evaluation setup I'd put in place for any RAG system going to production:
Week 1: Generate 60 synthetic samples with Ragas' TestsetGenerator. Manually review and trim to 40. Add 15 hand-written adversarial questions. Save as a versioned Parquet file.
Week 2: Run the full Ragas evaluation. Record baseline scores. Identify the 5 lowest-scoring samples. Fix the pipeline issue they reveal. Re-run. Confirm improvement.
Ongoing: Run a 10-sample smoke test in CI on every PR touching retrieval or prompt logic. Run the full 55-sample suite nightly. Alert if any metric drops more than 0.05 from the rolling 7-day average.
Monthly: Add 5–10 new test samples based on production queries that users flagged as wrong. Retire samples that test removed documentation. Regenerate synthetic samples if the corpus changed significantly.
The goal isn't a perfect score. It's knowing—within one CI run—whether your last change made the pipeline better or worse, and which component to investigate when it's worse. That's the difference between debugging a RAG system in twenty minutes and debugging it over a long weekend of print statements and intuition.