RePPL: Recalibrating Perplexity by Uncertainty in Semantic Propagation and Language Generation for Explainable QA Hallucination Detection

TL;DR

RePPL addresses hallucinations in large language models by recalibrating uncertainty from two sources: semantic-propagation uncertainty captured via token attribution across sampled generations (InnerPPL) and language-generation uncertainty via token confidences (OuterPPL). The two components are combined multiplicatively into a final RePPL score, yielding both strong detection performance and token-level explanations of hallucination triggers. Empirical results on four QA datasets across multiple instruction-tuned models show an average AUC of approximately 0.833 for all-datasets, outperforming several baselines and providing robust, explainable uncertainty signals. The work advances explainable QA hallucination detection and offers a practical, non-parametric approach suitable for zero-shot analysis with interpretable token-level cues.

Abstract

Large Language Models (LLMs) have become powerful, but hallucinations remain a vital obstacle to their trustworthy use. Previous works improved the capability of hallucination detection by measuring uncertainty. But they can not explain the provenance behind why hallucinations occur, particularly in identifying which part of the inputs tends to trigger hallucinations. Recent works on the prompt attack indicate that uncertainty exists in semantic propagation, where attention mechanisms gradually fuse local token information into high-level semantics across layers. Meanwhile, uncertainty also emerges in language generation, due to its probability-based selection of high-level semantics for sampled generations. Based on that, we propose RePPL to recalibrate uncertainty measurement by these two aspects, which dispatches explainable uncertainty scores to each token and aggregates in Perplexity-style Log-Average form as a total score. Experiments show that it achieves the best comprehensive detection performance across various QA datasets on advanced models (average AUC of 0.833), and it is capable of producing token-level uncertainty scores as explanations of hallucination.

Figures (9)

• Figure 1: Illustration of our motivation. The LLM (Llama-3.1-8B-Instruct) generates a hallucinated answer "Jascha Stoever" to the question "Who owns Perplexity?", with high-confidence (low-uncertainty) logits. This makes the detection hard for existing methods using logits. While our method judges it correctly with the explanation, by balancing uncertainty in semantic propagation and language generation.
• Figure 2: Detailed illustration of RePPL computing process. $(1)\sim(7)$ correspond to the same equations in paper. 'InnerPPLs' and 'OuterPPLs' with plural form to indicate token-wise confidence scores.
• Figure 3: Length of greedy ('Greedy') outputs and average length of sampled generations ('Sample') of 1000 examples in TriviaQA. 'Halu' is hallucinated ones, and 'No-Halu' is the not hallucinated ones.
• Figure 4: Performance under different hyperparameters. They are (a) our rescaling hypermeter group $(\alpha, \epsilon)$, (b) number of Generation $\mathbf{N}$, (c) temperature $\mathcal{T}$, (d) sampling hyperparameter group $(\text{Top-P}$, $\text{Top-K})$.
• Figure 5: Comparison of base attribution strategies.