Faithfulness-Aware Uncertainty Quantification for Fact-Checking the Output of Retrieval Augmented Generation

TL;DR

The paper introduces FRANQ, a faithfulness-aware uncertainty quantification framework for detecting factual errors in Retrieval-Augmented Generation (RAG). FRANQ first estimates the faithfulness of each claim to the retrieved evidence and then applies distinct UQ strategies for faithful and unfaithful cases, enabling targeted handling of grounding versus knowledge-based errors. It decomposes the true factuality of a claim as $P(c \text{ is true}) = P_{\text{faithful}}(c,\mathbf{r}) \cdot P(c \text{ is true} \mid \text{faithful}) + (1 - P_{\text{faithful}}(c,\mathbf{r})) \cdot P(c \text{ is true} \mid \text{unfaithful})$, and operationalizes this with AlignScore for faithfulness, Parametric Knowledge for unfaithful cases, and task-specific baselines for faithful cases, all calibrated via isotonic regression. A new long-form QA dataset annotated for both factuality and faithfulness supports evaluation, alongside four short-form QA datasets. Across multiple LLMs and tasks, FRANQ consistently improves factuality detection over baselines, demonstrating strong robustness to retrieval noise and offering practical, compact calibration suitable for real-world deployment. The work advances reliable RAG systems and lays groundwork for generation-time controls and better verification in knowledge-intensive applications.

Abstract

Large Language Models (LLMs) enhanced with retrieval, an approach known as Retrieval-Augmented Generation (RAG), have achieved strong performance in open-domain question answering. However, RAG remains prone to hallucinations: factually incorrect outputs may arise from inaccuracies in the model's internal knowledge and the retrieved context. Existing approaches to mitigating hallucinations often conflate factuality with faithfulness to the retrieved evidence, incorrectly labeling factually correct statements as hallucinations if they are not explicitly supported by the retrieval. In this paper, we introduce FRANQ, a new method for hallucination detection in RAG outputs. FRANQ applies distinct uncertainty quantification (UQ) techniques to estimate factuality, conditioning on whether a statement is faithful to the retrieved context. To evaluate FRANQ and competing UQ methods, we construct a new long-form question answering dataset annotated for both factuality and faithfulness, combining automated labeling with manual validation of challenging cases. Extensive experiments across multiple datasets, tasks, and LLMs show that FRANQ achieves more accurate detection of factual errors in RAG-generated responses compared to existing approaches.

Figures (15)

• Figure 1: franq illustration. Left: A user poses a question, and the RAG retrieves relevant documents and formulates an answer, potentially using information from the retrieved documents. Middle: The RAG output is decomposed into atomic claims. Right: The franq method assesses factuality by evaluating three components: (1) faithfulness, (2) factuality under faithful condition, and (3) factuality under unfaithful condition.
• Figure 2: Comparison of franq condition-calibrated with different choice of $\text{UQ}_\text{faith}$ and $\text{UQ}_\text{unfaith}$.
• Figure 3: Prompt used in short-form QA datasets. Titles and retrievals correspond to the Wikipedia page title and the passage retrieved from it.
• Figure 4: Prompt used in long-form QA datasets. Retrievals corresponds to the Wikipedia passage retrieved for input question.
• Figure 5: Prompt template used with GPT-4o for decomposing an answer into a set of atomic claims.