Why RAGs Hallucinate: Penalty-Aware Evaluation of Retrieval-Augmented Generation Systems with Knowledge-Gap Canaries
TL;DR Highlight
Scoring wrong answers at -4 and adding KB-absent questions exposed big differences in RAG answer-versus-decline policy.
Who Should Read
Backend developers whose internal-document RAG chatbot confidently answers questions the documents don't cover
Core Mechanics
- Accuracy alone makes RAG systems look the same
- Accuracy when answering was 97.0–98.0% for all three systems.
- The difference was whether they answer when they shouldn't.
- Plain accuracy rewards guessing
- If wrong answers cost 0, guessing beats declining.
- The paper scores correct +1, wrong −4, abstain 0, so answering only pays off at 80%+ confidence.
- Canaries catch answers from outside the documents
- A canary is a question verified to have no answer in the knowledge base (KB).
- Any answer counts as a violation, even a correct one, since it came from the model's memory.
- Canary violation rates differed sixfold
- Gemini RAG answered 98.1% of canary trials, CustomGPT.ai RAG 16.7%, OpenAI RAG 22.2%.
- Gemini had the highest accuracy (0.968) but dropped in penalty-aware ranking.
- Hallucination is mostly an answer-or-decline policy
Evidence
- Canary violation 16.7% vs 98.1%
- Based on 18 canaries from SimpleQA-Verified's 1,000 questions, 3 repeats, 54 trials.
- CustomGPT.ai 16.7%, OpenAI 22.2%, Gemini 98.1%.
- Accuracy and penalty-aware rankings flipped
- Accuracy: Gemini 0.968, OpenAI 0.948, CustomGPT.ai 0.866.
- Q (k=4): OpenAI +0.862, Gemini +0.793, CustomGPT.ai +0.767.
- Abstention rates varied over fiftyfold
- CustomGPT.ai 11.1%, OpenAI 3.3%, Gemini 0.2%.
- Accuracy when attempting was 97.5%, 98.0%, 97.0% respectively.
- Higher penalty pushes Gemini down
- At k=9 (90% confidence), Q was OpenAI +0.757, CustomGPT.ai +0.645, Gemini +0.588.
- OpenAI ranked first at every k from 1 to 9.
How to Apply
- Add 10 canary questions to your RAG eval set
- Pick questions you've verified aren't answerable from your KB and measure how often the bot declines.
- Log any answer as a violation, even a correct one, if it came from outside the KB.
- Recompute your metric with +1/−4/0
- Split your logs into correct, wrong, and abstain, then compute Q = correct rate − 4 × wrong rate.
- Try k = 1, 2, 4, 9 and check whether the ranking changes.
- Make empty-retrieval behavior explicit
- If there are zero citations or chunks, force a 'not found in documents' reply in the prompt and code.
- Tally wrong answers by whether evidence was attached
- Separate wrong answers with sources from those without, and check whether the answer was in the retrieved context to split retrieval from generation failures.
Try It
# Paste into Claude/ChatGPT (simplified version of the paper's method: canary check)
You are a document-based Q&A assistant. Answer only when the {{document}} below supports it; otherwise reply exactly "I can't confirm this from the document." Do not use general knowledge.
[Document]
{{document}}
[Question]
{{a question the document does not answer}}
Next, also ask a question {{question}} that the document does answer, and score by hand: correct +1, wrong −4, decline 0.Terminology
Related Papers
The Judge Can Debias Itself: Awareness and Self-Debiasing for Reliable LLM-as-a-Judge
Warning an LLM judge about bias or letting it self-critique improved 20 of 21 model–bias combinations.
90% of US working hours are spent on tasks not covered well by AI benchmarks
Mapping 29 benchmarks to job tasks shows 90% of US working hours lack good coverage
Postgres 19 delayed after AI reviewers find more bugs than humans can fix
LLM code review surfaced so many bugs that Postgres 19 slipped to November and 8-10 features were reverted
Show HN: Gemini 4 Argon plays StarCraft
LLMs wrote StarCraft bots in one hour; GPT-6 Astra scored highest.
A Plausible Explanation Is Enough to Mislead Multimodal Judges
Adding plausible explanations to fixed answers cut MLLM judges' wrong-answer rejection by 12.7–13.8 points.
Neural networks are clean
Looking at neurons directly shows most have one main job, challenging superposition.
Related Resources
Original Abstract (Expand)
Volume-based accuracy rewards retrieval-augmented generation (RAG) systems for guessing: a system that answers everything outscores one that declines when its knowledge base cannot support an answer. Building on the confidence-target analysis of Kalai et al. (2025), we present a penalty-aware evaluation framework for deployed RAG products, combining (i) asymmetric scoring (correct +1, wrong -4, abstain 0), (ii) knowledge-gap canaries, questions whose answers are verifiably absent from the knowledge base, so that any answer constitutes ungrounded generation from parametric memory, and (iii) a failure-attribution pipeline that separates retrieval, generation, and abstention-policy failures. Applying the framework to three commercial RAG systems and a no-retrieval baseline on SimpleQA-Verified (1,000 questions x 3 repeats, graded blind by a cross-family three-judge panel with 98.9% unanimity), we find that accuracy when answering is closely clustered across systems (97.0-98.0%), while canary violation rates differ roughly sixfold (16.7% vs. 98.1%). The systems are separated less by what they answer correctly than by whether they answer at all when they should not, and penalty-aware scoring reorders the volume-based ranking accordingly; the reordering is stable across penalty settings from k=1 to k=9. All code, configurations, transcripts, and judge votes are released for independent audit.