To Believe or Not to Believe Your LLM

TL;DR

This work addresses the challenge of hallucinations in large language models by separating epistemic and aleatoric uncertainty through an information-theoretic framework that leverages iterative prompting. It introduces a computable mutual-information lower bound on epistemic uncertainty, expressed as a function of a pseudo joint distribution over multiple model responses, and provides a finite-sample estimator with missing-mass guarantees. A score-based abstention mechanism is proposed, calibrated automatically to detect when the model’s output is unreliable, and semantic equivalences are incorporated to improve robustness. Empirically, the MI-based approach outperforms first-order uncertainty metrics on datasets with mixed single-label and multi-label queries, while maintaining competitive performance on simpler tasks, demonstrating practical utility for reliable AI-assisted systems.

Abstract

We explore uncertainty quantification in large language models (LLMs), with the goal to identify when uncertainty in responses given a query is large. We simultaneously consider both epistemic and aleatoric uncertainties, where the former comes from the lack of knowledge about the ground truth (such as about facts or the language), and the latter comes from irreducible randomness (such as multiple possible answers). In particular, we derive an information-theoretic metric that allows to reliably detect when only epistemic uncertainty is large, in which case the output of the model is unreliable. This condition can be computed based solely on the output of the model obtained simply by some special iterative prompting based on the previous responses. Such quantification, for instance, allows to detect hallucinations (cases when epistemic uncertainty is high) in both single- and multi-answer responses. This is in contrast to many standard uncertainty quantification strategies (such as thresholding the log-likelihood of a response) where hallucinations in the multi-answer case cannot be detected. We conduct a series of experiments which demonstrate the advantage of our formulation. Further, our investigations shed some light on how the probabilities assigned to a given output by an LLM can be amplified by iterative prompting, which might be of independent interest.

Figures (6)

• Figure 1: Single-label queries with low epistemic uncertainty: Conditional normalized probability of the correct completion given repetitions of an incorrect response. Each figure shows the query and the considered two responses with their initial probabilities, as a response for the query, in parentheses (the first response is the correct one).
• Figure 3: Multi-label queries with aleatoric uncertainty: Conditional normalized probability of the first of the two provided responses, both of which are correct, given repetitions of the second response in the prompt. Each figure shows the query and the considered two responses with their initial probabilities, as a response for the query, in parentheses.
• Figure 4: A hallucination: $\widetilde{Q}$ places an excessive mass where the ground truth $\widetilde{P}$ has a low mass.
• Figure 5: PR-curve for the baseline and the proposed methods on various datasets. On the TriviaQA and AmbigQA datasets, M.I. and S.E. perform nearly identically, but they outperform the $T0$ and S.V. baselines. For the S.E. and M.I. methods, the responses for a large number of queries can be clustered into a single group, and therefore the semantic entropy and mutual information scores are zero. This is why the starting point of their curves is at a higher recall values. On the TriviaQA+WordNet and AmbigQA+WordNet datasets with a significant number of high entropy multi-label queries, M.I. outperforms the S.E. baseline. The methods perform nearly identical on the recall area that is not shown.
• Figure 6: Recall and error rates (one minus precision: percentage of mistakes when not abstaining) of the proposed and the baseline method on TriviaQA+WordNet and AmbigQA+WordNet datasets. On TriviaQA+WordNet and AmbigQA+WordNet datasets, the methods are calibrated at target loss of $0.05$ and $0.15$, respectively. On the x-axis, the queries are partitioned according to the entropy of the LLM's output. Error bars show 2 standard deviation confidence intervals (based on 10 repetitions). While the first-order S.E. method has similar recall and error rates to those of the proposed M.E. method on low-entropy queries, its recall values are nearly zero for queries with higher entropy.

Theorems & Definitions (20)

• Definition 4.2: Pseudo joint distribution
• Remark 4.3: Sampling from $\widetilde{Q}$
• Definition 4.4: Epistemic uncertainty metric
• Theorem 4.5
• Theorem 4.6
• proof : Proof of \ref{['thm:MI']}
• Theorem E.1: Concentration of a missing mass berend2013concentration
• Theorem E.2
• Corollary E.3: Expected missing mass of Zipf distribution
• proof