Cycles of Thought: Measuring LLM Confidence through Stable Explanations

TL;DR

This work tackles the problem of unreliable uncertainty estimates in large language models (LLMs) by introducing stable explanations, a framework that treats the distribution over test-time explanations as a set of classifiers and computes a posterior over answers via marginalization. Explanations are sampled and filtered for logical consistency with entailment, enabling a Bayesian-like posterior $\widehat{p}(a|q) = \sum_e p_\phi(a|e,q)\,p(e|q)$. Across five QA datasets and two LLMs (GPT-3.5 and GPT-4), the approach improves selective uncertainty metrics (AUROC and AURC), with particularly strong gains on complex tasks like MMLU Law, while calibration remains comparable to or slightly behind baselines in some cases. The method offers a principled way to quantify uncertainty that adapts test-time decision boundaries via explanations and suggests connections to transductive inference and stability analyses, highlighting a path toward more trustworthy LLM deployments in high-stakes settings.

Abstract

In many high-risk machine learning applications it is essential for a model to indicate when it is uncertain about a prediction. While large language models (LLMs) can reach and even surpass human-level accuracy on a variety of benchmarks, their overconfidence in incorrect responses is still a well-documented failure mode. Traditional methods for ML uncertainty quantification can be difficult to directly adapt to LLMs due to the computational cost of implementation and closed-source nature of many models. A variety of black-box methods have recently been proposed, but these often rely on heuristics such as self-verbalized confidence. We instead propose a framework for measuring an LLM's uncertainty with respect to the distribution of generated explanations for an answer. While utilizing explanations is not a new idea in and of itself, by interpreting each possible model+explanation pair as a test-time classifier we can calculate a posterior answer distribution over the most likely of these classifiers. We demonstrate how a specific instance of this framework using explanation entailment as our classifier likelihood improves confidence score metrics (in particular AURC and AUROC) over baselines across five different datasets. We believe these results indicate that our framework is both a well-principled and effective way of quantifying uncertainty in LLMs.

Figures (12)

• Figure 1: Empirical distribution of explanation log likelihoods (top left) and explanation entailment probabilities (top right) generated for the TruthQA dataset using token logits from GPT3.5-Turbo. Red denotes explanations generated by conditioning on the incorrect answer and blue denotes explanations justifying the correct answer. While mean likelihood for the two explanation distributions are different, there is significant overlap. In contrast the tail of the incorrect explanation distribution is distinct when using entailment probability. The example explanation (lower) suggests we can use this entailment measure to distinguish semantically unlikely explanations in cases where likelihood fails.
• Figure 2: Comparision of LLM Confidence Elicitation Strategies. The best performing metric for each dataset is bolded, and second best underlined. (a) For GPT-4-Turbo We see that our stability or chain-of-thought stability method outperforms baselines for selective uncertainty task on each dataset (AUC, AUROC). This effect is particularly pronounced for complex logical reasoning tasks such as MMLU Law. (b) We also see on GPT-3.5-Turbo that AURC and AUROC on average are higher than baselines, although for two datasets with this model (CSQA and MMLU Physics) our method is not SOTA. ECE is highlighted in red as this evaluation can be misleading ding2020revisiting, but still include for transparency (see \ref{['sec:discussion']} for discussion).
• Figure 3: Stable Explanation Prompts
• Figure 4: Token Probability Prompt
• Figure 5: Linguistic Confidence Prompt