Enhancing Uncertainty Estimation in LLMs with Expectation of Aggregated Internal Belief

TL;DR

This work tackles the challenge of unreliable uncertainty estimation in large language models, particularly after RLHF, which can cause overconfidence and hallucinations. It introduces EAGLE, a training-free calibration method that leverages internal layer representations during self-evaluation, aggregating layer-wise logits from the last $k$ Transformer layers and computing the final confidence as the expectation over a distribution of candidate scores. Empirically, EAGLE significantly improves calibration (lower ECE) and discrimination (higher AUROC) across open-ended and multiple-choice tasks on multiple models (e.g., Llama3, Qwen2.5) and datasets (TriviaQA, GSM8K, MMLU), with ablations highlighting the benefits of last-n layer aggregation, logits-level aggregation, and score-distribution expectation. The approach demonstrates robustness to prompt variations and score granularity, offering a practical, training-free path to safer, more reliable uncertainty estimation in diverse LLM deployments.

Abstract

Large Language Models (LLMs) have achieved remarkable success across a wide range of natural language tasks, but often exhibit overconfidence and generate plausible yet incorrect answers. This overconfidence, especially in models undergone Reinforcement Learning from Human Feedback (RLHF), poses significant challenges for reliable uncertainty estimation and safe deployment. In this paper, we propose EAGLE (Expectation of AGgregated internaL bEief), a novel self-evaluation-based calibration method that leverages the internal hidden states of LLMs to derive more accurate confidence scores. Instead of relying on the model's final output, our approach extracts internal beliefs from multiple intermediate layers during self-evaluation. By aggregating these layer-wise beliefs and calculating the expectation over the resulting confidence score distribution, EAGLE produces a refined confidence score that more faithfully reflects the model's internal certainty. Extensive experiments on diverse datasets and LLMs demonstrate that EAGLE significantly improves calibration performance over existing baselines. We also provide an in-depth analysis of EAGLE, including a layer-wise examination of uncertainty patterns, a study of the impact of self-evaluation prompts, and an analysis of the effect of self-evaluation score range.

Figures (2)

• Figure 1: The EAGLE pipeline. First, the LLM generates an answer. Then, EAGLE prompts the LLM to self-evaluate the correctness of its answer, extracts the hidden states corresponding to the self-evaluation token from multiple layers, projects these hidden states to logits, and aggregates them. Finally, a softmax function is applied to the aggregated logits to compute the final confidence score.
• Figure 2: Layer-wise calibration performance on GSM8K for Llama3 8B and Qwen2.5 7B. The x-axis and y-axis represent the end and start layers for hidden state aggregation, respectively. Darker colors in ECE heatmaps and brighter colors in AUROC heatmaps indicate better calibration.