LLM Evaluation Framework

Abstract

A mature evaluation framework must therefore examine the entire pipeline rather than assign a single score to the final response. This paper presents an integrated evaluation methodology covering retrieval coverage, retrieval precision, ranking quality, context quality, answer relevance, faithfulness, factual correctness, citation attribution, retrieval-depth stability, scalability, robustness, latency, cost, and real-world task success.

The framework incorporates traditional information-retrieval metrics, the reference-free RAGAS methodology, the RAGGED framework’s RAG Stability Score and RAG Scalability Coefficient, LLM-as-a-Judge evaluation, human review, offline benchmarking, online experimentation, and post-deployment monitoring. Special attention is given to high-risk applications such as healthcare, finance, law, and public administration, where authority, provenance, timeliness, auditability, and justified refusal are more important than linguistic fluency alone.

Keywords: Retrieval-Augmented Generation, RAG evaluation, retrieval quality, faithfulness, groundedness, RAGAS, RAGGED, retrieval depth, hallucination, LLM-as-a-Judge, online monitoring

1. Introduction

Retrieval-Augmented Generation systems usually contain two broad components:

A retrieval component, which searches an external corpus for information relevant to a user query.
A generation component, which uses the query and retrieved material to construct a natural-language response.

In production systems, the architecture is more accurately represented as:

Knowledge ingestion→Indexing→Query processing→Retrieval→Reranking→Context integration→Generation→Citation and validation

This expanded pipeline introduces many distinct failure modes. For example, a system may retrieve the correct document but place it below the model’s effective attention range. It may retrieve a passage containing the correct fact alongside several contradictory or outdated passages. The model may produce a factually correct answer from memory but fail to use the retrieved evidence. Alternatively, it may remain perfectly faithful to a retrieved document that is itself incorrect or obsolete.

Consequently, evaluating only whether the final response “sounds good” is inadequate. A production-grade system must answer three fundamental questions:

Did retrieval find the right evidence?
Did generation use that evidence faithfully and answer the actual question?
Is the overall system stable, sustainable, auditable, and useful under real operating conditions?

These questions correspond to three major levels of evaluation:

Retrieval evaluation→Generation evaluation→System evaluation

RAGAS formalizes several reference-free dimensions of retrieval and generation quality, while RAGGED extends evaluation to the way performance changes as retrieval depth increases. The original RAGAS paper was published at EACL 2024, and the expanded RAGGED work appeared in the ICML 2025 proceedings.

2. Why a Comprehensive RAG Evaluation Framework Is Necessary

Enterprise RAG systems generally pursue two goals:

Higher accuracy, especially by reducing unsupported statements and hallucinations.
Greater stability, so that responses remain reproducible and useful across queries, documents, model versions, and retrieval configurations.

Many evaluations focus only on the visible answer. They ask whether it is readable, intelligent, or human-like. Those qualities matter, but they do not reveal where the answer came from or whether the system can be trusted.

A mature evaluation must diagnose at least three dimensions.

2.1 Retrieval quality

The system must retrieve sufficient, relevant, authoritative, and current evidence. If retrieval fails to obtain the necessary information, a stronger generator cannot reliably manufacture the missing facts.

This leads to an important engineering principle:

The lower bound of a RAG system is usually determined by retrieval.

2.2 Generation consistency

The answer must remain faithful to the evidence provided. It should not invent facts, contradict the retrieved material, silently combine incompatible sources, or present uncertain inferences as established facts.

2.3 System stability and sustainability

The system must continue to work when:

retrieval depth changes;
the corpus grows;
documents contain noise;
queries become ambiguous;
the model or embedding version changes;
users ask questions that the corpus cannot answer;
latency and cost constraints are introduced.

Evaluation therefore acts not merely as a scoreboard but as a validation of the team’s mental model of the system.

3. Layer One: Retrieval Evaluation

Retrieval evaluation establishes whether the system supplies the generator with usable evidence. It should measure coverage, noise, ranking, and the relationship between retrieved documents and the facts required to answer the query.

3.1 Constructing retrieval ground truth

Traditional retrieval metrics require a labelled evaluation set. Each test case should contain:

a user query;
one or more relevant documents or passages;
graded relevance labels where possible;
the evidence required to answer the query;
document version and effective date;
a reference answer when available;
an indication of whether the question is answerable from the corpus.

A binary relevance label is sometimes sufficient:

rel(d,q)∈{0,1}

However, graded relevance is more informative:

rel(d,q)∈{0,1,2,3}

where, for example:

0 means irrelevant;
1 means topically related;
2 means useful supporting evidence;
3 means directly contains the required answer.

3.2 Recall@K

Recall@K measures how much of the complete relevant-document set appears among the first K retrieved results:

Suppose five documents are required to support a multi-part answer and four appear in the top ten results:

Recall@K is especially important when:

answers require several sources;
the task involves multi-hop reasoning;
missing one document may remove an essential qualification;
regulations or policies contain distributed evidence.

A common simplified explanation says that Recall@K is “the proportion of correct documents among the top K.” That description is technically closer to precision. The denominator of recall must be the total number of relevant documents, not simply K.

High recall is valuable, but it should not be maximized without constraint. Returning more documents may improve coverage while simultaneously introducing more noise.

3.3 Hit Rate@K

Hit Rate@K asks whether the retrieved set contains at least one relevant document.

For one query:

Across N queries:

Hit Rate is useful for single-fact question answering but does not measure:

how many relevant documents were found;
where the first relevant document was ranked;
whether enough evidence was retrieved for a complete answer;
how much irrelevant content was returned.

3.4 Precision@K

Precision@K measures the proportion of the first K retrieved documents that are relevant:

If six of the first ten results are relevant:

Precision@10=0.6

Precision represents the retriever’s ability to filter noise. Low precision may produce:

unnecessary token consumption;
longer prompts;
higher inference costs;
attention dilution;
conflicting evidence;
greater hallucination risk;
reduced answer quality.

Precision and recall must be interpreted jointly. Aggressive filtering may raise precision while excluding evidence and reducing recall. Increasing K may improve recall but lower precision.

3.5 Error Discovery Rate

Some engineering materials use Error Discovery Rate, or EDR, to represent the proportion of irrelevant documents in the first K results:

Under binary relevance:

EDR@K=1−Precision@K

EDR is not a universally standardized RAG metric, so every evaluation report should state its precise definition.

3.6 Mean Reciprocal Rank

Mean Reciprocal Rank, or MRR, evaluates how early the first relevant result appears:

Here, rankᵢ is the rank of the first relevant document for query i.

Examples:

relevant document at rank 1: 1.0;
rank 2: 0.5;
rank 5: 0.2.

MRR is useful when the system primarily needs one decisive document, such as:

FAQ retrieval;
policy lookup;
customer-support knowledge search;
direct fact retrieval.

Its limitation is that it ignores all relevant documents after the first one.

3.7 Normalized Discounted Cumulative Gain

Normalized Discounted Cumulative Gain, or nDCG, incorporates graded relevance and document position.

The result is normalized against the ideal ranking:

nDCG rewards systems that place highly relevant documents near the top while assigning less credit to useful documents ranked lower.

It is preferable to MRR when:

multiple documents are relevant;
relevance has different degrees;
document order affects downstream generation;
the first result alone does not provide sufficient evidence.

4. Evaluating the Retrieved Context

Document-level retrieval metrics do not fully determine whether the context sent to the model is suitable for answer generation. A document may be broadly relevant while its selected chunk is not. Conversely, a passage labelled non-gold may still contain useful indirect clues.

Context evaluation should therefore examine relevance, precision, recall, redundancy, conflict, and ordering.

4.1 Context Relevance

Context Relevance measures whether the retrieved context contains information needed to answer the question and avoids unnecessary material.

The original RAGAS approach asks an LLM to extract sentences that are important for answering the query. The score is then calculated as:

This metric penalizes redundant context and represents the degree to which retrieval remains focused.

Low Context Relevance can be caused by:

poor embeddings;
incorrect metadata filters;
unsuitable chunk sizes;
poor query rewriting;
excessively broad retrieval;
weak reranking;
retrieval of topically similar but factually irrelevant passages.

4.2 Context Precision

Context Precision evaluates how much of the supplied context contributes to answering the question and, in ranking-aware versions, whether useful passages appear before irrelevant passages.

A simplified version is:

A more informative implementation should assign contribution labels at the passage or claim level and reward earlier placement of useful evidence.

Low Context Precision indicates that the generator is being asked to distinguish signal from excessive noise.

4.3 Context Recall

Context Recall measures whether the retrieved context covers the facts required by the reference answer:

This differs from document-level Recall@K:

For example, a retrieved document may be labelled relevant but omit one critical condition from the reference answer. Document recall may appear satisfactory while Context Recall remains incomplete.

4.4 Context conflict and version consistency

A production evaluation should also measure whether the retrieved set contains:

contradictory facts;
superseded policies;
duplicate passages;
documents from incompatible time periods;
evidence from different jurisdictions;
sources with different authority levels.

These conditions cannot be fully represented by similarity scores. They require metadata-aware rules, temporal validation, source ranking, or human review.

5. Layer Two: Generation Consistency Evaluation

Once evidence has been retrieved, the next question is whether the model uses it correctly.

The generation stage is particularly vulnerable to:

Hallucination: the model introduces unsupported or fabricated content.
Faithfulness failure: the response does not accurately reflect the supplied evidence.
Answer mismatch: the response may discuss the correct topic without answering the actual question.
Incomplete synthesis: the answer omits material facts or limitations.
Citation mismatch: citations are present but do not support the associated claims.

5.1 Faithfulness

Faithfulness measures whether the factual claims in the answer can be derived from the retrieved context.

Let the generated answer be decomposed into a set of independently verifiable claims:

Let V⊆S be the claims supported by the context. Then:

The original RAGAS method uses an LLM to:

break the answer into individual statements;
determine whether each statement is supported by the context;
calculate the proportion of supported statements.

This claim-level method is more appropriate than treating the complete answer as a single semantic vector.

A low score may reveal:

unsupported claims;
hallucinated names, dates, or numbers;
excessive inference;
reliance on parametric memory rather than retrieved evidence;
conflict between the answer and the cited material.

Faithfulness should not be confused with objective correctness:

If the retrieved source is outdated or false, the model may be highly faithful but factually wrong.

5.2 Groundedness

Groundedness is often used as a near-synonym for faithfulness. It asks whether every material conclusion in the answer has identifiable support in the evidence.

A strong groundedness evaluator should operate at the claim level:

It should ideally return not only a score but also:

unsupported claims;
supporting passage IDs;
entailment confidence;
contradictory evidence;
missing citations.

5.3 Factual Consistency

Factual Consistency evaluates whether the answer preserves the meaning and facts of the source material.

An LLM judge or natural-language inference model may classify each claim as:

entailed;
contradicted;
not enough information.

This extends faithfulness by explicitly detecting contradiction rather than merely checking semantic similarity.

5.4 Embedding-based screening

Some engineering pipelines convert answer sentences and knowledge passages into embeddings, calculate cosine similarity, and flag sentences below a threshold such as 0.65 as potential hallucinations.

This can be useful as a fast triage mechanism:

embed each answer sentence;
embed candidate evidence passages;
calculate the highest answer-to-passage similarity;
flag sentences below a selected threshold.

However, embedding similarity is not a sufficient measure of faithfulness. Two sentences may be semantically similar while disagreeing on:

a number;
a date;
a negation;
a patient group;
a jurisdiction;
an outcome;
a causal relationship.

Embedding thresholds should therefore be used for screening, not as the final factuality decision.

5.5 Answer Relevance

Answer Relevance evaluates whether the response directly addresses the user’s question.

It considers:

directness;
completeness;
avoidance of irrelevant details;
alignment with the requested task;
usefulness to the user.

It does not necessarily measure factual correctness.

The original RAGAS method asks an LLM to generate several possible questions from the answer, embeds those generated questions, and compares them with the original query:

Here, q is the original question and qᵢ is a question inferred from the generated answer.

A response can therefore have:

high relevance but low faithfulness;
high faithfulness but low relevance;
high correctness but poor completeness;
fluent language but little practical value.

These dimensions should not be collapsed prematurely.

5.6 Answer Correctness

Answer Correctness compares the generated response with an authoritative reference answer or verified facts.

It may combine:

factual agreement;
semantic similarity;
key-point coverage;
contradiction penalties;
numerical accuracy;
entity consistency.

Correctness and faithfulness must remain separate:

Situation Faithfulness Correctness
Answer follows an incorrect retrieved document High Low
Answer uses model memory and happens to be correct Low High
Answer is fully supported and objectively correct High High
Answer is unsupported and wrong Low Low

5.7 Completeness

Completeness measures whether the answer covers all material information required by the query.

An answer may be factually correct but incomplete if it omits:

one part of a multi-part question;
exceptions;
eligibility criteria;
dates;
risks;
limitations;
uncertainty;
required actions.

A useful formulation is:

5.8 Fluency and readability

Fluency evaluates:

grammatical quality;
natural language;
logical flow;
structural clarity;
absence of repetition;
appropriate level of detail.

Fluency matters for user experience, but it must never be treated as a substitute for correctness or groundedness. Highly fluent hallucinations are often more dangerous because users are more likely to trust them.

5.9 Attribution and citation sufficiency

Attribution evaluates whether factual claims identify their sources.

A basic metric is:

Citation correctness should be evaluated separately:

For high-risk systems, answers should retain stable evidence identifiers such as:

The policy became effective on 1 April 2025. [Doc12]

An auditable system should store:

claim text;
citation ID;
passage text;
source document;
version;
publication date;
retrieval timestamp;
document authority;
entailment result.

6. RAGAS: Reference-Free Evaluation of RAG Pipelines

RAGAS was introduced as a framework for automated, reference-free evaluation of RAG systems. Its original paper emphasizes three core dimensions:

Faithfulness: whether the answer is grounded in the retrieved context.
Answer Relevance: whether the answer addresses the question.
Context Relevance: whether the retrieved context is focused and useful.

The framework was designed to reduce dependence on manually annotated reference answers and support faster evaluation cycles.

6.1 Faithfulness in RAGAS

The answer is decomposed into claims. An LLM then determines which claims are supported by the context:

6.2 Answer Relevance in RAGAS

Questions are generated from the answer and compared semantically with the original user query:

This penalizes incomplete, indirect, or redundant answers.

6.3 Context Relevance in RAGAS

The evaluator extracts only the sentences necessary to answer the question:

This penalizes unnecessary context.

6.4 Limitations of automated RAGAS evaluation

RAGAS depends on another model for evaluation, so its scores may vary because of:

judge-model bias;
prompt sensitivity;
API model updates;
long-context limitations;
domain knowledge gaps;
nondeterministic outputs;
preference for particular writing styles.

The RAGAS study found that context relevance was harder to evaluate than faithfulness and answer relevance, especially for longer contexts.

Automated evaluation should therefore be calibrated against human judgments.

7. RAGGED: Evaluating Retrieval-Depth Stability and Scalability

Traditional RAG evaluation often tests one fixed top-K configuration. However, system performance may vary substantially when the number of retrieved passages changes.

Retrieval depth, denoted by k, is the number of retrieved passages supplied to the reader model.

RAGGED evaluates how different retriever-reader combinations behave across retrieval depths, datasets, domains, and noise conditions. It introduces two complementary metrics:

RAG Stability Score;
RAG Scalability Coefficient.

The framework’s central finding is that reader robustness to noise can be more important than retriever quality alone. The same retriever may help one reader architecture while harming another.

7.1 Retrieval is not always beneficial

RAG is frequently assumed to outperform closed-book generation, meaning generation without retrieved context. RAGGED demonstrates that this assumption is not universally valid.

In the paper’s tested configurations:

some reader models consistently benefited from additional context;
others peaked at a small k and then declined;
some models performed worse than their no-context baseline when exposed to noisy passages;
multi-hop questions generally required more evidence than simple single-hop questions.

These findings support the use of a closed-book baseline in every RAG experiment:

A negative value means retrieval has degraded performance.

7.2 Two retrieval-depth behaviours

Improve-then-plateau

The model improves as more context is added and eventually reaches a plateau:

This pattern indicates that the reader can continue extracting signal while tolerating additional noise.

Peak-then-decline

The model improves initially, reaches an early optimum, and then degrades:

This suggests sensitivity to irrelevant or distracting information.

RAGGED found that retrieval-depth behaviour was strongly reader-dependent. In its experiments, models such as Flan and GPT-3.5 often followed improve-then-plateau patterns, while tested LLaMA and Claude configurations were more vulnerable to noise at larger k. These are empirical findings from the tested models and datasets, not universal laws for every future model version.

7.3 RAG Stability Score

The RAG Stability Score, or RSS, measures how stable performance remains near the optimal retrieval depth.

Let:

S(k) is performance at retrieval depth k; k* is the retrieval depth with optimal performance; and Δ defines a local window around k*.

A conceptual representation is:

A value close to 1 indicates that small changes around the optimal k do not substantially affect performance.

A low score indicates that the system is brittle: selecting a retrieval depth only slightly different from the optimum may produce a major performance decline.

The RAGGED paper uses a symmetric window as a general diagnostic while recognizing that under-retrieval and over-retrieval may have asymmetric effects.

7.4 RAG Scalability Coefficient

The RAG Scalability Coefficient, or RSC, measures how much sustained benefit a system obtains as retrieval depth increases before performance plateaus or declines.

Conceptually, it evaluates accumulated improvement over a useful range of k:

where k
sat
​

is the final retrieval depth before the marginal gain falls below a predefined threshold or performance begins to decline.

A high RSC means that the model continues benefiting from more retrieved information across a broad range. A low RSC means that it:

plateaus early;
gains very little;
or becomes worse when more context is supplied.

RSS and RSC answer different questions:

A model can be stable but not scalable. For example, performance may remain consistently mediocre across several retrieval depths. It is therefore necessary to evaluate peak performance, RSS, and RSC together.

7.5 Retriever quality versus reader robustness

A stronger retriever does not automatically yield a better final answer.

RAGGED compares sparse and dense retrieval methods and finds that:

retriever performance varies by domain;
improved recall may not translate directly into better answer quality;
reader sensitivity to noise strongly affects downstream results;
specialized domains may benefit significantly from relatively small retrieval improvements;
reranking may help in open-domain tasks but may not consistently help specialized technical domains.

The framework also reports that reranking generally provided more reliable noise reduction than simply prompting the model to “focus on relevant information,” although their benefits did not necessarily accumulate when combined.

8. Additional Robustness and Operational Metrics

A complete RAG evaluation should move beyond retrieval relevance and answer quality.

8.1 Noise robustness

Noise Robustness evaluates whether the system remains accurate when irrelevant or misleading passages are added.

Test cases may include:

unrelated documents;
topically related distractors;
outdated policies;
contradictory evidence;
duplicate chunks;
truncated passages;
low-authority sources.

A possible score is:

A score near 1 indicates limited degradation.

8.2 Negative rejection

Negative Rejection measures whether the system refuses to answer when the corpus lacks sufficient evidence.

A reliable RAG system should distinguish:

answerable questions;
partially answerable questions;
conflicting-evidence questions;
unanswerable questions.

Forcing an answer in every case may improve superficial answer rates while increasing hallucination risk.

8.3 Counterfactual robustness

Counterfactual Robustness evaluates whether the system is misled by deliberately altered evidence.

Examples include:

changing a date;
reversing a conclusion;
replacing a drug indication;
modifying a financial figure;
changing a policy jurisdiction;
inserting a false causal relationship.

A system that merely copies retrieved content may fail this test. High-risk RAG therefore requires source authority, version control, and cross-source verification in addition to answer faithfulness.

8.4 Diversity

Diversity may be relevant when the task requires multiple perspectives, sources, or solution paths.

Possible dimensions include:

source diversity;
topic diversity;
institution diversity;
temporal diversity;
document-type diversity.

Diversity must be balanced with authority. Ten weak sources should not outweigh one definitive regulatory document.

8.5 Latency

Latency should be measured at multiple levels:

retrieval latency;
reranking latency;
model generation latency;
end-to-end latency;
P50, P95, and P99 latency.

RAG quality improvements that make the application unusably slow may not create real business value.

8.6 Cost and efficiency

Cost evaluation should include:

embedding cost;
retrieval infrastructure cost;
reranker cost;
prompt-token cost;
output-token cost;
judge-model evaluation cost;
human-review cost.

Useful operational metrics include:

and:

9. Traditional NLP Metrics

Traditional NLP metrics remain useful but should be treated as supporting indicators.

9.1 Exact Match

Exact Match checks whether the generated answer exactly equals a reference answer.

It is suitable for:

short factual answers;
extractive question answering;
highly standardized outputs.

It is overly strict when several phrasings are valid.

9.2 Token-level F1

Token-level F1 calculates the harmonic mean of token precision and recall between prediction and reference.

It provides partial credit when wording differs slightly.

9.3 BLEU

BLEU evaluates n-gram overlap and is traditionally associated with machine translation. It is less reliable for open-ended RAG answers because correct answers can use different wording.

9.4 ROUGE

ROUGE measures overlap with a reference and is commonly used for summarization. It can help assess content coverage but cannot reliably determine groundedness or factual accuracy.

9.5 Semantic similarity and BERTScore

Embedding-based or contextual similarity metrics can recognize paraphrases. However, they may overlook small but critical factual differences.

Traditional metrics cannot independently answer:

whether the answer is supported by retrieval;
whether a citation entails a claim;
whether a source is current;
whether the system should have refused to answer.

10. LLM-as-a-Judge and Human Evaluation

10.1 LLM-as-a-Judge

An LLM evaluator can assess:

faithfulness;
relevance;
correctness;
completeness;
citation support;
clarity;
refusal quality;
safety.

Common judging formats include:

numerical scoring;
pass/fail decisions;
pairwise comparison;
rubric-based multi-dimensional scoring;
claim-level entailment assessment.

Pairwise comparison often produces more stable judgments than asking the model to assign an absolute score without calibration.

LLM judges may nevertheless exhibit:

position bias;
verbosity preference;
self-preference;
sensitivity to prompt wording;
domain limitations;
inconsistent reasoning;
model-version drift.

The evaluation model, prompt, temperature, and rubric should therefore be version-controlled.

10.2 Human evaluation

Some dimensions still require human judgment, particularly in finance, healthcare, legal services, and public policy.

Human reviewers should evaluate:

factual correctness;
completeness;
fluency;
usefulness;
attribution;
source authority;
timeliness;
compliance;
appropriateness of refusal.

A strong human-review protocol uses stratified sampling rather than purely random sampling. Samples should include:

high-risk questions;
low-scoring answers;
high-scoring answers for false-positive checks;
newly added documents;
model-update cases;
user complaints;
citation mismatches;
unanswerable questions;
conflicting-evidence cases.

Inter-rater agreement should also be measured.

11. Three-Stage Evaluation Lifecycle

A RAG system should pass through three evaluation stages.

11.1 Stage One: Offline evaluation

Offline evaluation occurs before deployment using a labelled benchmark.

A standard workflow includes:

preparing query-answer pairs;
identifying relevant documents and supporting passages;
executing retrieval;
generating answers;
calculating retrieval, context, generation, and system metrics;
comparing candidate configurations;
analyzing failure categories.

Metrics may include:

The goal is to detect visible problems such as:

retrieval bias;
chunking errors;
poor metadata filters;
weak reranking;
excessive hallucination;
incomplete answers;
unstable retrieval depth.

11.2 Stage Two: Grey or A/B evaluation

A/B testing exposes a new configuration to a portion of real traffic.

It should compare:

user satisfaction;
human ratings;
task-completion rate;
citation usage;
answer abandonment;
follow-up question rate;
escalation to a human;
response latency;
cost per interaction.

A/B testing answers a critical question:

Does an offline metric improvement produce real user or business value?

A configuration may improve Context Recall but generate excessively long answers. Another may improve faithfulness while reducing helpful synthesis. Online experimentation reveals these trade-offs.

11.3 Stage Three: Online monitoring

Deployment does not end evaluation.

Continuous monitoring should track:

retrieval-empty rate;
no-answer rate;
citation coverage;
citation accuracy;
unsupported-claim rate;
answer refusal rate;
user negative feedback;
P50/P95/P99 latency;
model errors;
token usage;
cost;
document-version distribution;
knowledge-base freshness;
model and prompt drift.

Production monitoring transforms RAG from a laboratory prototype into an operational system.

12. Evaluation for High-Risk Industries

Healthcare, finance, law, insurance, and government require stricter standards.

The central questions are not whether the answer sounds intelligent, but whether:

information has been fabricated;
claims are traceable;
sources are authoritative;
sources are current;
applicable rules and jurisdictions are correct;
uncertainty is disclosed;
insufficient evidence triggers refusal;
the output can be audited.

12.1 Retrieval-level validation

The corpus should consist of controlled, approved sources rather than unrestricted web content.

Each document should retain:

source organization;
document ID;
publication date;
effective date;
expiry date;
version;
jurisdiction;
authority level;
access permissions;
approval status.

12.2 Generation-level validation

Material claims should preserve citation IDs, for example:

The company reported operating income of X for the fiscal year. [Doc12]

A stronger system maps every factual sentence to one or more evidence spans.

12.3 Audit-level validation

Human reviewers should inspect whether:

the citation supports the claim;
the correct document version was used;
dates from different reporting periods were mixed;
a management statement was presented as fact;
a guideline was applied outside its intended population;
a conclusion omitted a critical limitation;
the system should have refused to answer.

One inaccurate sentence in a medical or financial report may materially alter a decision. In such environments, the objective is not “generate more fluently,” but “produce an answer that is demonstrably correct and traceable.”

13. Metric Trade-offs

No RAG metric should be optimized in isolation.

13.1 Recall versus precision

Very high recall may introduce noise. Very aggressive precision optimization may remove critical evidence.

The preferred operating point depends on the task:

13.2 Constraint versus expressiveness

Over-constrained generation may:

sound mechanical;
omit helpful synthesis;
reduce readability;
fail to connect evidence.

Overly free generation may:

hallucinate;
overgeneralize;
introduce unsupported conclusions;
weaken auditability.

A practical rule is:

factual claims require evidence;
analysis must be labelled as analysis;
inference must be distinguished from fact;
insufficient evidence should lead to uncertainty or refusal.

13.3 Peak performance versus stability

A configuration with the highest score at one exact value of k may be less deployable than a slightly weaker but more stable configuration.

Therefore, model selection should consider:

The weights should reflect business risk rather than convenience.

13.4 Automation versus human review

Automated metrics provide scale and speed. Human review provides domain judgment and accountability.

The correct design is not to choose one, but to create a hierarchy:

Automated screening→LLM evaluation→Risk-based human review

14. A Recommended Production Evaluation Matrix

Layer Metric Question answered
Retrieval Recall@K Were all required documents retrieved?
Retrieval Hit Rate@K Was at least one relevant document found?
Retrieval Precision@K How much retrieved content is relevant?
Retrieval MRR How early does the first relevant result appear?
Retrieval nDCG Are highly relevant documents ranked near the top?
Context Context Relevance Is the context focused on the question?
Context Context Precision How much context contributes to the answer?
Context Context Recall Does the context cover every required fact?
Generation Faithfulness Are answer claims supported by the context?
Generation Answer Relevance Does the response answer the user’s question?
Generation Correctness Is the response objectively accurate?
Generation Completeness Are all required elements covered?
Generation Attribution Are factual claims properly cited?
Generation Citation Accuracy Do citations support their associated claims?
Robustness Noise Robustness Does irrelevant context degrade performance?
Robustness Negative Rejection Does the system refuse unanswerable questions?
Robustness Counterfactual Robustness Can false evidence mislead the system?
Retrieval depth RSS Is performance stable near optimal k?
Retrieval depth RSC Does performance continue improving as k grows?
Operations Latency Is the response fast enough?
Operations Cost Is the system economically sustainable?
Business Task Success Does the user complete the intended task?
Business User Satisfaction Is the result useful in practice?
Governance Traceability Can every important claim be audited?

15. Recommended End-to-End Evaluation Procedure

A production RAG evaluation can follow the sequence below.

Step 1: Build a representative benchmark

Include:

common questions;
long-tail questions;
multi-hop questions;
ambiguous questions;
unanswerable questions;
temporal questions;
adversarial questions;
high-risk questions.
Step 2: Establish ground truth

For each query, record:

relevant documents;
evidence spans;
relevance levels;
required facts;
reference answers;
acceptable alternative answers;
required refusal conditions.
Step 3: Evaluate retrieval

Calculate:

Recall@K, Precision@K, HitRate@K, MRR, nDCG

Test several K values rather than one fixed setting.

Step 4: Evaluate context construction

Measure:

Context Relevance;
Context Precision;
Context Recall;
duplication;
contradiction;
temporal consistency;
source authority.
Step 5: Evaluate generation

Measure:

Faithfulness;
Answer Relevance;
Correctness;
Completeness;
Citation Coverage;
Citation Accuracy;
refusal appropriateness.
Step 6: Evaluate stability and scalability

Run the system across a range of retrieval depths:

k∈{1,2,3,5,10,20,…}

Calculate or inspect:

Inject:

irrelevant passages;
outdated documents;
contradictory claims;
counterfactual facts;
incomplete questions;
prompt-injection content.
Step 8: Calibrate automated judges

Compare LLM scores with expert human labels. Revise:

rubrics;
prompts;
thresholds;
sampling strategies.
Step 9: Run A/B tests

Confirm that offline gains improve real task success without unacceptable cost or latency.

Step 10: Monitor continuously

Version and record:

retriever;
embedding model;
reranker;
reader model;
prompt;
knowledge base;
evaluation dataset;
metric implementation.

16. Discussion

A qualified RAG evaluation report should not merely state:

Recall@10 is 0.86 and Faithfulness is 0.91.

It should explain:

RAG evaluation is therefore better understood as a test of system understanding.

It reveals whether the development team understands:

where knowledge enters the system;
how retrieval errors propagate;
how the model distinguishes signal from noise;
how claims are connected to evidence;
when the system should refuse;
how errors can be reproduced and corrected.

The goal is not to maximize every number. It is to build a controlled and explainable system whose limitations are visible.

17. Conclusion

A reliable RAG system must do more than retrieve documents and produce fluent text. It must:

find the necessary evidence;
rank useful evidence appropriately;
avoid excessive noise;
cover the facts required to answer the question;
remain faithful to the evidence;
distinguish fact from inference;
cite claims accurately;
reject unsupported questions;
remain stable as retrieval conditions change;
continue delivering value after deployment.

Traditional retrieval metrics such as Recall@K, Precision@K, MRR, and nDCG remain essential. RAGAS contributes automated evaluation of faithfulness, answer relevance, and context relevance. RAGGED adds retrieval-depth stability and scalability through RSS and RSC. Human evaluation, robustness testing, A/B experiments, and online monitoring complete the production lifecycle.

The central philosophy can be summarized as follows:

RAG evaluation is not only about whether a system can produce an answer. It is about whether the system can find the facts, remain faithful to those facts, explain where they came from, and take responsibility for what it says.

References

Es, S., James, J., Espinosa Anke, L., and Schockaert, S. RAGAs: Automated Evaluation of Retrieval Augmented Generation. Proceedings of EACL 2024, System Demonstrations, pp. 150–158.
Hsia, J., Shaikh, A., Wang, Z. Z., and Neubig, G. RAGGED: Towards Informed Design of Scalable and Stable RAG Systems. Proceedings of ICML 2025, PMLR 267, pp. 24139–24155.
Yu, H., Gan, A., Zhang, K., Tong, S., Liu, Q., and Liu, Z. Evaluation of Retrieval-Augmented Generation: A Survey. The survey organizes RAG evaluation around retrieval and generation outputs, relevance, accuracy, faithfulness, datasets, benchmarks, and limitations.