RAG Evaluation

RAG evaluation is the disciplined process of testing whether a retrieval-augmented generation system finds the right evidence, uses that evidence faithfully, answers the user’s question, and continues to perform reliably after deployment. A company uses it to compare chunking, embedding, search, prompt, model, and knowledge-base changes before those changes reach employees or customers. For example, an internal HR assistant can be tested on policy questions to verify that it retrieves the current leave policy, gives an answer supported by that policy, and refuses to invent an answer when the policy does not cover the question. RAG evaluation is therefore broader than checking whether a chatbot sounds plausible. (docs.langchain.com)

What RAG evaluation measures

A RAG system has at least two quality boundaries:

1. Retrieval: Did the system find useful, sufficiently complete, and correctly ranked evidence?
2. Generation: Did the model produce an accurate, relevant, complete, and evidence-grounded response?

A production evaluation adds a third boundary:

3. Operations: Is the system fast enough, affordable enough, permission-aware, observable, and safe to operate?

Microsoft’s RAG evaluators distinguish process evaluation of retrieval from system evaluation of the final response. Ragas similarly provides separate metrics for context precision, context recall, faithfulness, response relevancy, and related dimensions. (docs.ragas.io)

The core RAG metrics

Layer	Metric or test	What it asks	Typical failure revealed
Retrieval	Context precision	Are relevant chunks ranked above distracting chunks?	Search returns many near-matches or stale documents
Retrieval	Context recall	Did retrieval include the evidence needed for the expected answer?	Correct information exists but was not retrieved
Retrieval	Document retrieval or recall@k	Did the system return the labeled relevant document within the top-k results?	Embedding, filters, query rewriting, or ranking problem
Generation	Faithfulness or groundedness	Are the answer’s claims supported by the supplied context?	Hallucinated details or unsupported extrapolation
Generation	Response relevance	Does the answer address the user’s question directly?	Evasive, generic, or off-topic output
Generation	Completeness or answer correctness	Does the response cover the important parts of the reference answer?	Partial answer, omitted condition, or wrong conclusion
Operations	Latency, cost, refusal, permissions	Does the workflow behave within business constraints?	Slow responses, excessive judge cost, data leakage, or unsafe guessing

The metric names are not perfectly standardized across tools. Microsoft Foundry, for example, exposes groundedness, relevance, response completeness, retrieval, and document retrieval evaluators, while Ragas groups related concepts under its own metric library. Choose metrics according to the decision you need to make rather than treating one vendor’s score as a universal standard. (docs.ragas.io)

Build an evaluation dataset before tuning the pipeline

A useful dataset is not just a list of easy questions generated from documents. It should represent the questions the system is expected to answer, the questions it must decline, and the ways users are likely to phrase requests.

Include:

• Common questions: Frequently requested policies, procedures, product details, or account instructions.
• Paraphrases: The same intent expressed with different terminology.
• Ambiguous questions: Requests that need clarification before retrieval can be trusted.
• Multi-hop questions: Questions requiring evidence from more than one document.
• Negative cases: Questions for which the knowledge base has no approved answer.
• Freshness cases: Questions where a newer policy should outrank an archived version.
• Permission cases: Questions where the answer differs by user, team, region, or account.
• Adversarial cases: Prompt injection, misleading wording, conflicting documents, and malformed input.

Each example should capture the query, expected answer or answer criteria, relevant source identifiers, user or permission context where applicable, and the retrieved passages and final response produced by each experiment. LangSmith describes the basic loop as creating a dataset, running the RAG application on that dataset, and applying evaluators to the outputs. (docs.langchain.com)

A small, carefully labeled regression set is usually more useful for engineering decisions than a large, weakly labeled collection. Start with cases that represent business risk and user volume, then expand it from real failure reports.

A practical evaluation workflow

1. Freeze the system configuration

Record the document-ingestion version, chunking rules, embedding model, vector or hybrid search settings, reranker, prompt, generation model, filters, and answer policy. Without this metadata, a score change cannot be connected to a specific system change.

2. Test retrieval independently

Run the query set through retrieval without asking the language model to answer. Inspect whether the correct source appears, where it ranks, whether chunks contain enough surrounding context, and whether stale or unauthorized documents are excluded.

For labeled retrieval tests, use document identifiers or relevance judgments where possible. For unlabeled tests, use a rubric-based evaluator carefully and manually review borderline cases.

3. Test answer generation with captured context

Evaluate the final answer against both the retrieved context and, where available, a reference answer. This separates two common problems:

• The answer is wrong because the right evidence was never retrieved.
• The evidence was available, but the model misunderstood it, omitted a condition, or added unsupported claims.

A grounded answer is not automatically a complete or useful answer. Conversely, an answer that resembles a reference answer may still be unsafe if its claims are not supported by the context shown to the model.

4. Add deterministic checks

LLM judges are useful for nuanced language, but deterministic tests are valuable for rules that should not be subjective. Check for required citations, source IDs, forbidden phrases, refusal behavior, JSON schema validity, escalation labels, permission filters, maximum context size, and latency thresholds.

5. Compare experiments, not isolated scores

Use the same dataset and evaluation configuration when comparing a new embedding model, reranker, chunking strategy, prompt, or generation model. Review score deltas by question type, not only the overall average. A change that improves common policy questions but breaks permission-sensitive cases may be unacceptable even if its aggregate score rises.

How to interpret evaluation results

Do not use a single threshold as a substitute for judgment. Thresholds should reflect the consequence of failure. A customer-facing answer about account eligibility may require stricter review than an internal brainstorming assistant. For regulated, financial, medical, employment, or security-sensitive workflows, include human review and explicit escalation rules rather than relying only on automated scores.

LLM-as-a-judge: useful, but not self-validating

LLM-based evaluators can assess semantic correctness, relevance, groundedness, and completeness at a scale that is difficult to achieve with exact-match rules. Ragas notes that some metrics use one or more LLM calls, and LangSmith supports code evaluators, LLM-as-judge evaluators, composite evaluators, and evaluators aligned to labeled feedback. (docs.langchain.com)

Their limitations matter:

• The judge can misunderstand the reference answer or the source context.
• A fluent but incorrect answer may receive an overly generous score.
• Scores can vary with the judge model, prompt, temperature, and rubric.
• Judge calls add latency and usage cost.
• Sensitive company data may need to remain within an approved environment.

Calibrate evaluators against a human-reviewed sample. Define what counts as a pass, partial pass, and failure. Review disagreements and update the rubric or dataset instead of silently trusting the score.

Cost drivers and operating model

RAG evaluation cost is driven by the number of test examples, the number of pipeline variants, retrieval and generation calls per example, judge-model calls, embedding or indexing work, trace storage, and the frequency of continuous evaluation. Ragas explicitly identifies LLM-based metrics as potentially requiring model calls, while evaluation platforms such as LangSmith store experiment outputs, scores, and traces for comparison. (docs.ragas.io)

A sensible operating model has three layers:

• Pull-request or pre-release checks: A small critical regression set with deterministic checks and selected judge metrics.
• Scheduled evaluation: A broader dataset run after knowledge-base, model, retrieval, or prompt changes.
• Production monitoring: Sampled traces, user feedback, escalation rates, refusal quality, latency, permission failures, and newly discovered questions.

Keep separate budgets for development experiments and recurring monitoring. Re-run expensive evaluations when the system changes materially, not necessarily after every low-risk content update.

When a RAG evaluation program is suitable

RAG evaluation is worthwhile when the system answers from internal documents, customer records, product information, policies, technical manuals, or other changing knowledge. It is especially important when users need traceable answers, when incorrect answers create operational or compliance risk, or when multiple teams are changing the knowledge base and model stack.

It is less useful to begin with a large evaluation framework when the underlying knowledge source is incomplete, contradictory, or unmanaged. First establish document ownership, versioning, access rules, and an ingestion process. The existing FollowAI guide on Document Ingestion Pipeline covers that upstream foundation, while Corporate AI Knowledge Base explains the broader knowledge-base system.

What FollowAI can build

FollowAI can design, code, connect, launch, operate, monitor, and improve a complete RAG evaluation system around your approved knowledge sources. That can include:

• Connectors for SharePoint, cloud storage, websites, databases, CRM records, ticketing systems, or internal applications.
• Versioned ingestion and indexing with document ownership, timestamps, source IDs, and permission metadata.
• A labeled evaluation dataset covering common, ambiguous, negative, multi-hop, freshness, and access-sensitive questions.
• Retrieval and answer evaluators for context precision, context recall, groundedness, relevance, completeness, refusal behavior, and business-specific rules.
• Experiment tracking for chunking, embeddings, hybrid search, reranking, prompts, models, and retrieval filters.
• Continuous monitoring of traces, user feedback, escalation, latency, cost, stale content, and permission failures.
• Approval gates before changing prompts, models, retrieval configuration, or published knowledge.

The continuously running parts can include scheduled indexing, regression evaluations after approved content changes, sampled production trace review, alerting on failed thresholds, and routing uncertain answers to a designated owner. Required or optional approval remains with your team for sensitive-source publication, threshold changes, access-policy changes, and high-risk answer escalation.

The result is one connected knowledge-base and evaluation operating system rather than a handoff between separate document developers, search specialists, CRM integrators, and automation contractors. FollowAI can also connect the evaluation output to the systems where remediation occurs—for example, opening a content-owner task when a policy answer fails, routing a permission issue to IT, or creating a review queue for unresolved user questions.