Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Retromorphic Testing with Hierarchical Verification for Hallucination Detection in RAG

Boxi Yu, Yuzhong Zhang, Liting Lin, Lionel Briand, Emir Muñoz

Abstract

Large language models (LLMs) continue to hallucinate in retrieval-augmented generation (RAG), producing claims that are unsupported by or conflict with the retrieved context. Detecting such errors remains challenging when faithfulness is evaluated solely with respect to the retrieved context. Existing approaches either provide coarse-grained, answer-level scores or focus on open-domain factuality, often lacking fine-grained, evidence-grounded diagnostics. We present RT4CHART, a retromorphic testing framework for context-faithfulness assessment. RT4CHART decomposes model outputs into independently verifiable claims and performs hierarchical, local-to-global verification against the retrieved context. Each claim is assigned one of three labels: entailed, contradicted, or baseless. Furthermore, RT4CHART maps claim-level decisions back to specific answer spans and retrieves explicit supporting or refuting evidence from the context, enabling fine-grained and interpretable auditing. We evaluate RT4CHART on RAGTruth++ (408 samples) and RAGTruth-Enhance (2,675 samples), a newly re-annotated benchmark. RT4CHART achieves the best answer-level hallucination detection F1 among all baselines. On RAGTruth++, it reaches an F1 score of 0.776, outperforming the strongest baseline by 83%. On RAGTruth-Enhance, it achieves a span-level F1 of 47.5%. Ablation studies show that the hierarchical verification design is the primary driver of performance gains. Finally, our re-annotation reveals 1.68x more hallucination cases than the original labels, suggesting that existing benchmarks substantially underestimate the prevalence of hallucinations.

Retromorphic Testing with Hierarchical Verification for Hallucination Detection in RAG

Abstract

Large language models (LLMs) continue to hallucinate in retrieval-augmented generation (RAG), producing claims that are unsupported by or conflict with the retrieved context. Detecting such errors remains challenging when faithfulness is evaluated solely with respect to the retrieved context. Existing approaches either provide coarse-grained, answer-level scores or focus on open-domain factuality, often lacking fine-grained, evidence-grounded diagnostics. We present RT4CHART, a retromorphic testing framework for context-faithfulness assessment. RT4CHART decomposes model outputs into independently verifiable claims and performs hierarchical, local-to-global verification against the retrieved context. Each claim is assigned one of three labels: entailed, contradicted, or baseless. Furthermore, RT4CHART maps claim-level decisions back to specific answer spans and retrieves explicit supporting or refuting evidence from the context, enabling fine-grained and interpretable auditing. We evaluate RT4CHART on RAGTruth++ (408 samples) and RAGTruth-Enhance (2,675 samples), a newly re-annotated benchmark. RT4CHART achieves the best answer-level hallucination detection F1 among all baselines. On RAGTruth++, it reaches an F1 score of 0.776, outperforming the strongest baseline by 83%. On RAGTruth-Enhance, it achieves a span-level F1 of 47.5%. Ablation studies show that the hierarchical verification design is the primary driver of performance gains. Finally, our re-annotation reveals 1.68x more hallucination cases than the original labels, suggesting that existing benchmarks substantially underestimate the prevalence of hallucinations.

Paper Structure

This paper contains 54 sections, 14 equations, 2 figures, 10 tables.

Table of Contents

  1. Introduction
  2. Preliminaries
  3. Approach
  4. Theoretical Foundation: Retromorphic Testing
  5. Pipeline Overview
  6. Claim Decomposition
  7. Local Verification
  8. Chunking.
  9. OR-join ($\mathcal{O}_{or}$).
  10. Chunk-wise local assessment.
  11. Global Verification
  12. Adaptive prompting strategy.
  13. Answer-Level Aggregation
  14. AND-join ($\mathcal{O}_{and}$).
  15. Experiments
  16. ...and 39 more sections

Figures (2)

  • Figure 1: Detection granularity on a motivating example. RT4CHART adds explicit context evidence to localized answer spans. Red denotes Contradicted, green Entailed, and yellow Baseless.
  • Figure 2: Overview of RT4CHART. The system decomposes $A$ into claims, verifies them locally and globally against $C$, and returns claim-level and answer-level outputs.