Estimating the Hallucination Rate of Generative AI

TL;DR

This paper addresses the problem of quantifying hallucination risk in in-context learning by casting CGMs as approximating the posterior predictive of an unknown Bayesian mechanism. It introduces the posterior hallucination rate (PHR) and a practical predictive-resampling estimator that leverages only log-probabilities from the CGM to approximate the likelihood-based thresholding that defines hallucinations. The authors provide a Doob-theorem–based justification and finite-N approximations, and validate the approach with synthetic regression and natural-language ICL experiments using Llama-2 and Gemma-2 models. Overall, the method offers a principled way to quantify and monitor epistemic/aleatoric uncertainty in ICL, with empirical evidence that PHR tracks the true or model-based hallucination rates across tasks, though calibration and underestimation biases remain practical considerations for complex settings.

Abstract

This paper presents a method for estimating the hallucination rate for in-context learning (ICL) with generative AI. In ICL, a conditional generative model (CGM) is prompted with a dataset and a prediction question and asked to generate a response. One interpretation of ICL assumes that the CGM computes the posterior predictive of an unknown Bayesian model, which implicitly defines a joint distribution over observable datasets and latent mechanisms. This joint distribution factorizes into two components: the model prior over mechanisms and the model likelihood of datasets given a mechanism. With this perspective, we define a hallucination as a generated response to the prediction question with low model likelihood given the mechanism. We develop a new method that takes an ICL problem and estimates the probability that a CGM will generate a hallucination. Our method only requires generating prediction questions and responses from the CGM and evaluating its response log probability. We empirically evaluate our method using large language models for synthetic regression and natural language ICL tasks.

Figures (26)

• Figure 1: An example of an in-context dataset and generated response examples for the last label. The correct response, "Sports," is displayed in green. Wrong answers are in purple.
• Figure 2: In the first and third panes, we see the neural process's generated outcomes for $n = 2$ and $n = 100$. The blue region is the true (1-$\epsilon$)--likely set, while the purple is the likely set when conditioned on the blue data points. The second and fourth panes are the corresponding measures of the PHR and THR across the domain.
• Figure 3: Synthetic data: (a) Plot of the average THR and PHR as a function of $n$. The THR and PHR follow each other and decrease as the number of contextual examples ${n}$ increases. (b) Plot of the estimated PHR vs the THR. Each point represents the predicted PHR and actual THR for a given instance. Perfect performance would have all points on the $x=y$. Results show the PHR closely approximates THR, but performance degrades with context length.
• Figure 4: Performance Metrics for Llama-2-7b Across Tasks.
• Figure 5: Llama-2-7b: Error Rate (Top green curves) and Response Entropy (Bottom blue curves) on LLM in-context learning tasks. Grey dashed lines represent the error rate and entropy of a random classifier over the set of valid responses.

Theorems & Definitions (12)

• Definition 1
• Definition 2
• Definition 3
• Definition 4
• Theorem 1: Doob's Informal
• Theorem 2: PHR via Posterior Predictive
• Lemma C.1
• proof
• Theorem 3: PHR via Posterior Predictive
• proof