RAG Hallucination Detection: Methods for Production
Your AI assistant quotes your documents. Here is how to detect the answers your documents don't support, before a customer does.
Key takeaways
- RAG hallucination detection means checking each answer against the passages the system looked up, and flagging any claim those passages don't support.
- Run a citation check on every live answer, a grounding check by a second model on most answers, and human review on a sample plus everything flagged.
- When a check fails, first ask whether the right passage was found at all. If it wasn't, the fix is in the search, and no prompt change will help.
- Score hallucination on its own, with a floor. An answer with one invented figure should fail even when its average score looks fine.
- A flagged answer should not reach a customer as written. Depending on what a wrong answer costs, label it, answer again from quoted passages, or hold it and pass the question to a person.
1.5%, the fund assistant tells an investor who asked about a fund's exit fee, with a link to the fund's fact sheet. The fact sheet says 2%. The case is made up, but it is a typical one: the answer sounded sure, and it even came with a source.
This is a hallucination: an AI stating something false as if it were true.
A RAG system (short for retrieval-augmented generation: an AI that looks up your documents before it answers) makes fewer of them, but not none. The ones it makes are dangerous, because they come with a source attached.
Detecting them takes a few checks on every live answer, and a clear next step when a check flags one.
What is RAG hallucination detection?
RAG hallucination detection means checking each answer from a RAG system against the passages it looked up, and flagging any claim those passages don't support. In production, run a cheap citation check on every answer, a grounding check by a second AI model on most, and human review on a sample.
The test for every answer:
Every claim in the answer should point to a passage that says it.
A claim the passages support is grounded. A claim they don't support fails the check and is flagged as a likely hallucination, even if it happens to be true. That matters for a business: you can only stand behind what your documents say.
What is hallucination in RAG, and does RAG reduce hallucinations?
The US standards body NIST calls it confabulation: "confidently stated but erroneous or false content" (NIST AI 600-1, July 2024). OWASP's list of AI risks describes content "that seems accurate but is fabricated" (OWASP LLM09:2025).
In a RAG system, it happens in one of two places:
- The search missed. The right passage was never found, so the model answered from whatever nearby text it was given.
- The model went beyond the passage. The right passage was there, and the model misread it, mixed it with another, or added something from memory.
So yes, RAG reduces hallucinations, but it doesn't remove them. Anthropic's guide to reducing hallucinations says its techniques "don't eliminate them entirely" and advises you to "always validate critical information". Detection is that validation.
Which hallucination detection methods work in production?
Use the cheapest check that can detect the problem, and stack them. AWS's guide to detecting hallucinations in RAG systems (May 2025) compares several methods on accuracy and cost. The question in production is simpler: which check runs on every answer, and which on a sample?

- Citation check, on every answer. Code checks that the answer names a source, that the source was really retrieved, and that every number in the answer appears in it. It is fast and free.
- Grounding check, on most answers. A second model, or a small detector model, checks every claim against the passages.
- Human review, on a sample. A person reads a fixed sample each week, small enough to read in one sitting, plus every answer a check flagged.
The methods at a glance
| Method | What it detects | Cost and speed | Where to run it |
|---|---|---|---|
| Citation check (code) | Missing source, a source that wasn't retrieved, a number not in the source | Free, instant | Every answer |
| Grounding check (LLM judge) | Claims the passages don't support, including reworded ones | One extra model call per answer | Most answers, or all where errors are costly |
| Detector model | Unsupported words, marked in the answer | Small and fast; needs testing on your documents | Every answer, if it holds up on your data |
| Consistency check | Answers that change when you ask again | Several extra model calls | Spot checks, high-risk questions |
| Human review | Anything, including what the checks missed | Slow and costly | A sample, plus every flagged answer |
Grounding checks and detector models
The grounding check is often called a faithfulness score. The Ragas docs define it as the share of an answer's claims "supported by the retrieved context". A second model doing this marking is an LLM as a judge, and it needs calibrating against a person before you trust it.
Detector models are newer. The LettuceDetect paper (February 2025) describes one that reads the passages, the question and the answer, and marks the unsupported parts of the answer down to single words. It reports 30 to 60 examples a second on one graphics card.
Was the hallucination from the search or the model?
When a check flags an answer, the fix depends on where the hallucination started. Ask one question first: was the passage that holds the right answer among the passages the system found?

- The right passage wasn't found. The model (the AI that writes the answer) had nothing correct to work from. Fix the search, the way documents are split, or how tables and images are read.
- The right passage was found, and the answer still differs. The model went beyond it. Tighten the instructions, ask for quotes before the answer, or test another model.
A better prompt can't fix a passage the search never found.
We learned this on India Avenue's fund assistant. Early answers were often wrong. Many key figures sat inside images of charts and tables, the search couldn't read them, and the assistant answered from nearby words. Our post on how to test a RAG chatbot tells that story in full.
To ask this question at all, you need to log which passages each answer was given. Without that record, every hallucination looks like a model problem.
How do you handle hallucinations in RAG when a check flags an answer?
A flagged answer should not reach the customer as written. What happens instead depends on how costly a wrong answer is.
| A wrong answer costs… | Example | What to do with a flagged answer |
|---|---|---|
| Little | An internal help bot on office policies | Show it with a warning and the source; log it for review |
| Some | A product question from a customer | Answer again, using only quoted passages; if it still fails, say it doesn't know |
| A lot | A fee, a return or a medical dose | Hold it; say what the documents do support, and pass the question to a person |
Anthropic's guide suggests the same move as the middle row: check each claim for a supporting quote, and if there isn't one, "it must retract the claim". Letting the assistant say "I don't know" is part of detection, not a failure of it. Our post on making AI agents say "I don't know" covers how to design that reply.
How we detect hallucinations in Avesta AgentOS
Avesta AgentOS is the platform we build and run AI agents on. Three patterns from it, and from IngestIQ and Metricsense, the products we run alongside it, are worth copying.
1. Keep the page with every passage
When our knowledge layer, IngestIQ, splits a document into passages, it stores the page number, page range and section title with each one. So every answer can point to the exact place a figure came from.
That is what makes a citation check possible: you can check that the cited page was retrieved and that it holds the number. Without it, a source is just a file name.
2. Score hallucination on its own, with a floor
Metricsense evaluates each new release at scale. A grader model scores each AI answer in the test batch on five things. One of them is hallucination: is anything in the answer fabricated? The overall score is the average of the five.
But the hallucination score also has its own floor. An answer below it is flagged as a hallucination failure that needs review, whatever the average says.
One invented figure should fail an answer, even when the rest of it is excellent.
When the answers it grades carry their own confidence score, the grader's instructions say to ignore it. A sure-sounding answer gets no credit for sounding sure.
3. Test on the client's own questions, on every change
For India Avenue, every figure must link back to an approved fund document, and the assistant says so when a figure isn't in the documents. Their team supplied and marked 50 real questions, and every change still runs against them (as of 21 Oct 2026).
Two of the checks are "Does the figure match the fund document?" and "Is the source link there?". Those detect hallucinations before release. The live checks above detect the ones a test set never asked about.
Which RAG systems don't need every hallucination check?
Not every RAG system needs every layer.
- A person reads every answer before it's used. A drafting tool for your own team already has human review. A citation check is enough.
- Nobody acts on the answer. An internal search helper that shows passages with a short summary is low risk. Log answers and review a sample.
- You could show the passage instead. If users only need to find the right page, return the page. No generated answer means nothing to hallucinate.
For a customer-facing assistant, keep the grounding check on most answers. Add human review of a sample when a wrong answer costs money, trust or a licence.
The bottom line
RAG hallucination detection is a set of checks that runs on every live answer. When a check flags an answer, find out whether the search or the model caused it before you change anything.
For the step before production, see how to test a RAG chatbot.
Can your logs show which passages each live answer was given? Every fix above starts there.
Want hallucination checks on every answer your AI agents give, with a person reviewing what matters?
Get started: hello@avestalabs.ai
Frequently asked questions
- What should a RAG system log for hallucination detection?
- For each answer, log the question, the passages the search returned, the answer, the source it cited and the result of every check. The passages matter most. Without them you can't tell whether the search missed the right passage or the model went beyond it, so every hallucination looks like a model problem and gets the wrong fix.
- What is a faithfulness score in RAG?
- It is the share of an answer's claims that the retrieved passages support. The Ragas library defines it this way: split the answer into claims, check each claim against the passages, then divide supported claims by total claims. A score of 1 means every claim is supported. It is a useful grounding check, but it can't tell you whether the search found the right passages.
- Can the model detect its own hallucinations?
- Partly. You can ask it to find a supporting quote for each claim after it answers, and to drop any claim it can't support. Don't trust its own confidence score, though. In our evaluation tools, when an answer carries a confidence score, the grader is told to ignore it and judge only what the answer says. A separate check is more reliable than self-assessment.
- Are there small models for hallucination detection?
- Yes. Detector models read the passages, the question and the answer, and mark the words, or parts of words, the passages don't support. The LettuceDetect paper (February 2025) reports a model about 30 times smaller than the best models it compares with, handling 30 to 60 examples a second on one graphics card. They are cheap enough to run on every answer, but test one on your own documents first.
Share this article




