How to Measure RAG Quality and Reduce Hallucinations

A RAG system can work correctly from a technical standpoint and still give wrong answers.

The documents have been indexed. Search works. The model receives context. The API returns a response. Everything looks fine — until a user asks: where did that answer come from?

And then it turns out the system found the wrong document. Or missed a key piece of information. Or received the correct context but added something that was not there. Or gave a factually correct answer that still failed to solve the user's problem.

All of these situations are often casually described as "RAG hallucinations." That is too broad a simplification. OpenAI describes hallucinations as plausible-sounding but false statements generated by a model. In a RAG system, however, a wrong answer can arise before the model starts generating text — at the data or retrieval layer.

That is why there is no single meaningful metric such as "our RAG is 92% accurate" that tells you whether the whole system works well. To diagnose RAG properly, you need to measure its layers separately.

A good RAG evaluation process should answer at least six questions. Is the correct and current information present in the knowledge base? Did the retriever find the right chunks? Did it find all the information needed to answer? Did the model answer only on the basis of the available evidence? Did the answer actually solve the user's question? And can the system refuse to answer when there is not enough information?

Only by combining these signals can you determine why an answer was wrong and what actually needs to be improved.

What does a "hallucination" actually mean in a RAG system?

In practice, it helps to separate at least four classes of failure. This is not one official, universally accepted RAG taxonomy. It is a practical diagnostic split that helps assign a problem to the right layer of the system.

A source data problem

The information needed to answer is missing from the corpus, or the available sources are outdated, incorrect, incomplete, or contradictory.

Imagine this situation. In a company, the current maximum discount a salesperson can grant without a director's approval is 10%, but the knowledge base contains only an old policy with a 15% limit. The retriever finds the 15% document. The model answers: "A salesperson can grant a maximum discount of 15%."

The answer is wrong from a business perspective. At the same time, you cannot say that the model or retrieval failed. The system correctly used the only source it was given. The problem is in the knowledge corpus.

In our BLUP-FLOCK system, we saw how strongly answer quality depends on what happens before any model is involved. When we changed the physical data collection process on the litter farm — new scales, a different workflow — the quality of the system's results improved without changing the model. No embedding model can fix what the source data does not capture correctly.

A retrieval problem

The correct information exists in the knowledge base, but the system fails to find it or ranks the results incorrectly.

Using the same example — the index contains both the old policy with 15% and the new one with 10%. The user asks about the maximum discount. The retriever returns the old document but not the current one. The model again answers 15%.

This time, the knowledge was available. Retrieval failed. This is where metadata — effective date, document status, version — can materially improve retrieval quality, provided the pipeline actively uses it for filtering, ranking, or source selection. Simply having a valid_from=2026 field changes nothing if the retriever ignores it. We discussed this in more detail in our article on preparing data for RAG.

An answer generation problem

The retriever finds the correct chunk: "A salesperson may grant a discount of up to 10% without approval." But the model answers: "The maximum discount is 15%."

The evidence was available, but the final claim is not supported by the context provided. This is the case most closely related to what we measure in RAG evaluation as faithfulness or groundedness. Ragas defines Faithfulness by checking what proportion of the claims in an answer are supported by the retrieved context.

The answer is correct but not useful

The user asks: "Can I terminate the contract today?" The system answers: "The rules for terminating the contract are described in section 12 of the policy."

The information may be completely true. It may be correctly grounded in a source. But the user still does not know whether they can terminate the contract.

This is not a factual correctness or groundedness problem. It is a task completion and answer relevance problem.

Why does this distinction matter?

Because each class of failure requires a different intervention. If the right document is not in the knowledge base, changing the system prompt will fix nothing. If the retriever failed to find the document, changing the generation model may also change nothing. If retrieval works very well but the model regularly adds information that is not in the context, then the problem really is closer to the generation layer.

A wrong answer is the result of the entire pipeline, not just the work of the LLM.

There is no single "RAG accuracy"

LangSmith separates RAG evaluation into correctness, relevance, groundedness, and retrieval relevance. Each evaluation compares different elements of the system — the answer against a reference answer, the answer against the question, the answer against the retrieved documents, or the documents against the question. Ragas takes a similar approach, offering metrics such as Context Precision, Context Recall, and Faithfulness.

In practice, it is useful to think in layers. At the retrieval layer, we ask: did we find the right information (retrieval relevance), are the important results ranked highly (Context Precision), and did we find everything we needed (Context Recall)? At the generation layer, we ask: does the answer follow from the context (Faithfulness/Groundedness), and does it actually answer the user's question (Answer Relevance)? End-to-end, we evaluate the final result (Answer Correctness). At the system level, we also look at when the system answers and when it abstains (coverage, error rate, abstention).

The key word is "helps." Low Answer Correctness does not automatically tell you which layer failed. The cause may be a bad source, poor retrieval, missing context, a grounding failure, or even a problem with the reference answer itself. Metrics are primarily diagnostic tools.

Context Precision — are the right chunks ranked highly?

Context Precision evaluates the ranking of retrieved chunks. Ragas describes it as a measure of whether a retriever places relevant chunks above irrelevant ones.

Suppose the system retrieves 10 chunks and two of them contain the answer. If they are ranked 1st and 2nd, the result looks much better than if they are ranked 9th and 10th. In both cases the system technically "found the answer." But the quality of retrieval is not the same.

This matters especially when only a limited number of top results are passed to the model. It also matters when too much additional text increases cost and makes the context harder to use — the "Lost in the Middle" effect we discussed in our previous article is very relevant here.

Context Recall — did we lose anything important?

Precision and recall answer two different questions. Precision: how much of what we found was useful? Recall: how much of what we needed did we manage to find?

Ragas notes that Context Recall requires some form of reference — a reference answer, reference contexts, or, in the ID-based variant, the identifiers of chunks that should have been retrieved.

A concrete example. The user asks: "What conditions must a customer meet to qualify for free service?" The correct answer requires three elements — an active contract, a service inspection completed within the last 12 months, and no outstanding payments. The retriever finds the first two conditions. The retrieved chunks may be highly relevant, so precision can still look good. But retrieval is incomplete. That is the type of problem recall helps expose.

Faithfulness does not mean correctness

This is one of the most important distinctions in RAG evaluation.

Suppose the system retrieves a document saying: "The notice period is 60 days." The model answers: "The notice period is 60 days." The answer matches the context perfectly. It has high groundedness. But the document is from 2023, while the current policy says 30 days. The answer is therefore faithful but incorrect.

Ragas measures Faithfulness as the degree to which claims in the answer are supported by the supplied context. LangSmith separates groundedness — answer versus retrieved documents — from correctness, where the answer is evaluated against a reference answer.

A high faithfulness score answers the question "did the model stick to the sources?", not "were the sources true?" Those are two different problems. That is why it matters so much that sources in the corpus are current and properly versioned — without that, even perfect grounding does not guarantee a correct answer.

Who evaluates the evaluator?

We have a metric — for example, Faithfulness. But we still have to decide how to measure it. This distinction is critical: the metric defines what we want to measure; the evaluator defines how we measure it.

An evaluator can be deterministic code, an ID comparison, a text similarity method, a business rule, a human reviewer, or an LLM-as-a-judge. LangSmith supports evaluation with code, humans, LLM-as-a-judge, and pairwise comparisons. Ragas illustrates the difference well with Context Precision, offering LLM-based variants, non-LLM methods based on text similarity, and ID-based variants.

Is an LLM-as-a-judge score "the truth"?

No. If the evaluator itself uses a language model, its output is an automated judgment generated according to defined criteria, not a directly observable fact.

LangSmith explicitly points out that LLM-as-a-judge requires reviewing results and tuning the evaluator prompt. Its documentation describes comparing judge ratings against expert annotations and iteratively improving agreement between the evaluator and human judgment.

That is why a result such as faithfulness = 0.91 should not be interpreted as "the system is objectively 91% faithful." First, you need to know how the metric was defined, which evaluator was used, what dataset it was calculated on, and how well the evaluator agreed with expert judgment.

Use deterministic checks where you can

If you have a hard reference, use it without involving an LLM unnecessarily. If we know that a query should retrieve the documents policy_2026 and discount_rules, we can compare their IDs against the retrieval results. We do not need a language model to determine whether the expected document was found.

LLM-as-a-judge is more useful when semantic judgment is needed — whether the answer actually solves the user's problem, whether a claim is supported by a passage phrased differently, or whether two differently worded answers are materially equivalent.

Without a good dataset, there is no good evaluation

You can build an excellent metrics dashboard and still know very little about RAG quality if you test it on the wrong set of questions.

LangSmith recommends starting with manually curated, high-quality examples that represent both typical scenarios and edge cases. Later, the dataset can be expanded with real production traces and cases where users reported problems.

For RAG, it is useful to collect several different types of ground truth.

Question plus expected answer — the simplest version. Question: "What maximum discount can a salesperson grant without a director's approval?" Reference: "10%." This allows you to evaluate final correctness, but it has limitations. Two answers — "10%" and "A salesperson may grant a discount of up to 10%" — are materially equivalent. That is why exact match is often insufficient for generative answers, and LangSmith points to LLM-as-a-judge as one method of semantic evaluation.

Question plus the correct sources — we record that the correct source for a given question is sales_policy_v4, section 7.2. This lets us test retrieval independently of generation. If the system answers incorrectly, we can see whether the right document was found, where it ranked, and whether it was passed to the model. This often provides more diagnostic value than comparing final answers alone.

Question plus expected system behavior — not every question should end with an answer. If a user asks about something that is not in the knowledge base, the expected behavior may be to abstain: "I cannot find this information in the available documentation." This type of case should also be part of the dataset. Otherwise, we mostly test questions that are answerable and learn nothing about how the system behaves when evidence is missing.

The dataset has to contain difficult cases

A real dataset should not consist only of simple questions such as "What is the warranty period?"

It should include situations where the answer is spread across several documents, similar documents exist for different products, old and new versions coexist, the user uses terminology that differs from the company's documentation, the question is ambiguous, sources contradict one another, there is no answer, or the correct response is to ask for clarification.

These edge cases are often what reveal whether the architecture is robust or simply performs well on demo questions.

We have seen this repeatedly in our agent projects. While building the Ads Agent, which operates across multiple Google Ads accounts at the same time, we encountered a case where the agent mixed data from different client accounts in a briefing — producing a coherent-sounding but incorrect analysis. On simple questions involving one account, everything looked fine. The problem only surfaced on a dataset covering multiple clients — exactly the kind of edge case that is easy to miss when testing only "happy path" scenarios.

You also need to measure "I don't know"

Measuring accuracy alone can reward a system for guessing more often.

OpenAI illustrated this problem with an evaluation in which a model more willing to abstain had slightly lower accuracy but a much lower error rate than a model that guessed more often. A wrong answer and a deliberate refusal to answer should not be treated as the same type of failure.

At the same time, a high abstention rate proves very little on its own. A system that answers "I don't know" to every question will probably generate very little false information. It will also be practically useless.

That is why it is worth separating at least four categories: correct answer (the system answered and was right), incorrect answer (it answered incorrectly), correct abstention (it correctly refused to answer), and unnecessary abstention (it refused despite having sufficient evidence).

From there, you can track answer coverage — the share of queries the system attempts to answer — and the error rate among the answers it does provide. This gives you a much more useful picture than a single "abstention rate."

Metrics should not be maximized independently

This is one of the most important challenges in designing RAG systems. In production, we are not optimizing precision, recall, and faithfulness in isolation. We also have to balance quality against coverage, latency, cost, and safety. A change to one part of the pipeline may improve one metric while making another aspect of the system worse.

Higher retrieval k — retrieving more candidates can increase the chance of finding the document you need. But a larger k also means more results to process. Importantly, increasing the number of documents retrieved is not the same as passing more chunks into the LLM prompt. These are different stages.

More context is not automatically better — additional chunks may provide useful evidence, but they can also introduce more noise, consume more of the context budget, increase cost, and add latency. The goal is not to send the model as many documents as possible. The goal is to provide enough relevant context for the specific task.

Chunk size is a trade-off — there is no universal value such as "the best chunk is 500 tokens." Microsoft notes that the right chunking strategy depends on document structure, query characteristics, and the models in use. Larger chunks may preserve broader context better, while smaller chunks may represent individual passages more precisely in search. These parameters should be compared on a real dataset.

Hybrid search is not a magic switch — vector search is strong at semantic similarity, while keyword search is especially useful for exact strings such as symbols, product numbers, names, dates, and specialist terminology. Azure AI Search combines both approaches using Reciprocal Rank Fusion. Microsoft reports relevance gains from hybrid retrieval with semantic ranking in its tests, but that does not mean every application will automatically perform better without its own evaluation.

Reranking improves ranking at the cost of another pipeline stage — the retriever can first find a larger candidate pool, and a separate stage can then rerank those results before they reach the generator. This separates candidate retrieval from final context selection. But reranking adds work to the pipeline, so its impact should be evaluated together with quality, latency, and cost.

More restrictive abstention can improve safety but reduce coverage — we can configure a system to answer only when the evidence is strong enough. But if the gating is too restrictive, the system will also refuse questions it could have answered correctly. There is no universal "safest threshold." The cost of a false answer in a system supporting a doctor is completely different from the cost of one in an internal documentation search tool.

A similarity score is not "answer confidence"

This is a common implementation mistake. A vector database returns a score related to similarity or document ranking. That does not mean a score of 0.87 represents an 87% probability that the answer is correct.

In Azure AI Search, different ranking mechanisms use different score scales — BM25, vector similarity, RRF, and the semantic ranker calculate scores differently and return values in different ranges.

That is why gating such as if similarity > 0.8: answer() should not be treated as a universal hallucination-control mechanism. A threshold should be tied to a signal that has been validated on data from that specific system — for example, a combination of retrieval quality, reranker score, the presence of required sources, evidence quality, and business rules.

What matters is not how neat the number looks, but whether it actually separates cases where the system should answer from cases where it should abstain on your dataset.

What should the system do when sources disagree?

This is one of the cases that absolutely belongs in testing. Suppose retrieval returns two documents — one says "the notice period is 30 days," the other says "60 days." The model should not simply guess which one is correct.

The architecture can use different strategies depending on the data available. Recency priority — if documents have reliable effective dates, the system can prefer the current document. Authority priority — an official policy may outrank an FAQ, an employee note, or a presentation. Conflict disclosure — the system can answer: "The available documents contain conflicting information: one states 30 days, while another states 60 days." Or abstention — in a high-risk use case, the correct response may be: "I cannot determine a definitive answer from the available sources."

Metadata, versioning, and source hierarchy help resolve some of these issues. They do not guarantee knowledge consistency. They are parts of an evidence-management system, not an automatic solution to data quality problems.

First, identify which layer is failing

The most common mistake when improving a RAG system looks like this: "The system answered incorrectly. Let's improve the prompt." The prompt is only one part of the pipeline.

A more useful diagnostic process starts with a simple decision tree.

Does the correct knowledge exist in the corpus? No — it is a data problem. Yes — move on.

Did the system find it? No — it is a retrieval problem. Yes — move on.

Did the correct chunk reach the generator? No — the problem is ranking or context selection. Yes — move on.

Does the answer follow from the evidence? No — it is a grounding failure. Yes — move on.

Is the answer actually correct and useful? No — check the sources, the ground truth, and how the task is being completed.

When evidence is missing, can the system avoid guessing? No — you need an abstention or gating mechanism.

Only this kind of diagnosis lets you make engineering decisions based on data rather than intuition.

In practice, it looks like this: low retrieval recall means it is worth checking chunking, query rewriting, hybrid retrieval, and metadata filtering. Good retrieval with weak groundedness suggests a problem with generation instructions, the model, or evidence formatting. High groundedness with low correctness should make you question source freshness and ground-truth quality. A correct but unhelpful answer is a task-completion and response-format problem. If the system guesses when evidence is missing, you need an abstention policy. If the system refuses too often, check whether false abstention is blocking valid answers.

This diagnostic process is more useful than any list of "10 ways to improve RAG." The same technique can be highly effective in one system and completely unnecessary in another.

Offline evaluation is not the end

Before deployment, we can test the system on a prepared dataset. But users will almost always start asking questions the product team did not anticipate. That is why production observability is also necessary.

LangSmith separates offline evaluation — performed on prepared examples, often with references — from online evaluation on real production traces, where a reference answer is usually unavailable.

In practice, it is worth logging at least the user's question, the processed query sent to the retriever, retrieved documents and chunks, result order and reranking scores, the context passed to the LLM, the final answer, the sources used in the answer, the answer-or-abstain decision, user feedback, and evaluator results.

This turns a wrong answer from a one-off event such as "AI made something up again" into a case that can be reproduced, assigned to a specific layer, and added to a regression dataset.

We build our agent systems with the same principle. In the Ads Agent, every proposed action — a budget change, pausing an ad, modifying a strategy — goes through an approve/reject pipeline where we store the recommendation, historical context, and the human decision. When something needs correction, we can trace the data the agent used to support its proposal. The same approach to observability works well in RAG.

What does a mature RAG evaluation process look like?

I would not start with the question, "Which eval library should we use?" I would start by defining failure modes — what counts as an error in this specific system? The definition is different for a knowledge search tool, a support bot, a legal system, an internal copilot, and a sales assistant.

Then build a representative dataset. Include typical questions, difficult questions, missing answers, source conflicts, and different ways of asking the same thing. Assign a metric to each important failure mode. Do not measure everything just because a library offers 20 metrics.

Choose an evaluator. If you have a hard reference, use it. If you need semantic judgment, consider LLM-as-a-judge. And validate the evaluator itself — especially if its score will influence decisions about changing the model or architecture.

Define the trade-offs. What matters most in this product — coverage, minimizing wrong answers, latency, cost, or completeness? Test changes regressively — a change to chunking, the retriever, the model, or the prompt should be compared on the same dataset. And feed production problems back into the dataset — online monitoring should provide new cases for future offline evals.

At that point, evaluation becomes part of the product development cycle rather than a one-time test before deployment.

What RAG metrics will not tell you

Metrics are necessary, but none of them can answer the question "Is our RAG good?" on its own.

Context Precision can show ranking quality, but it cannot tell you whether the sources are true. Context Recall can show whether retrieval missed necessary evidence, but it requires an appropriate reference. Faithfulness can show whether the answer stays within the context, but it cannot guarantee that the context is current. Answer Correctness can show disagreement with ground truth, but it does not tell you whether the failure came from the data, retrieval, or generation. LLM-as-a-judge can automate semantic evaluation, but its output also needs oversight and may require calibration against expert judgments. And the retriever's similarity score is not the probability that the final answer will be correct.

A mature RAG evaluation process is therefore not about finding one perfect metric. It is about building a process that lets you answer: What exactly failed? How do we know? Which change should improve it? And did that change actually improve the system without making another important part worse?

Only then does evaluation become an engineering tool rather than just a dashboard full of numbers.

See how we build production AI systems and LLM integrations →

Sources

• OpenAI - Why language models hallucinate.
• LangChain / LangSmith - Evaluate a RAG application.
• LangSmith - Evaluation concepts.
• LangSmith - Application-specific evaluation approaches.
• LangSmith - Improve LLM-as-a-judge evaluators using human feedback.
• Ragas - Context Precision.
• Ragas - Context Recall.
• Ragas - Faithfulness.
• Microsoft Learn - Chunk large documents for RAG and vector search in Azure AI Search.
• Microsoft Learn - Hybrid search using vectors and full text in Azure AI Search.
• Microsoft Learn - Relevance scoring in hybrid search using Reciprocal Rank Fusion.