Are LLM Decisions Faithful to Verbal Confidence?

TL;DR

This work scrutinizes whether LLMs translate verbal confidence into risk-aware actions by introducing RiskEval, which prompts models to answer or abstain under varying penalties $λ$ and reports a Bayes-optimal threshold $τ(λ) = \frac{λ}{1+λ}$. It reveals a persistent dissociation: models maintain calibrated verbal confidence yet rarely adjust abstention policies as risk rises, leading to utility collapse under high penalties, though post-hoc enforcement of the optimal policy $π^*$ can salvage some utility. Prompt engineering to elicit risk-sensitive abstention has little effect, indicating a deeper behavioral prior rather than signal degradation. The findings challenge the sufficiency of calibrated verbal confidence for trustworthy AI and point to the need for training or inference-time mechanisms that couple uncertainty signals to risk-aware decisions.

Abstract

Large Language Models (LLMs) can produce surprisingly sophisticated estimates of their own uncertainty. However, it remains unclear to what extent this expressed confidence is tied to the reasoning, knowledge, or decision making of the model. To test this, we introduce $\textbf{RiskEval}$: a framework designed to evaluate whether models adjust their abstention policies in response to varying error penalties. Our evaluation of several frontier models reveals a critical dissociation: models are neither cost-aware when articulating their verbal confidence, nor strategically responsive when deciding whether to engage or abstain under high-penalty conditions. Even when extreme penalties render frequent abstention the mathematically optimal strategy, models almost never abstain, resulting in utility collapse. This indicates that calibrated verbal confidence scores may not be sufficient to create trustworthy and interpretable AI systems, as current models lack the strategic agency to convert uncertainty signals into optimal and risk-sensitive decisions.

Figures (8)

• Figure 1: The RiskEval Framework. We evaluate strategic abstention by prompting models with varying error penalties ($\lambda$) ranging from 0 to 100. Although models successfully verbalize uncertainty, they fail to translate this signal into decision-making. As illustrated, abstention rates on the HLE benchmark hle2025 remain largely invariant to increasing penalties.
• Figure 2: Verbalized Confidence is Invariant to Risk. The flat trajectories show that internal uncertainty estimates remain stable despite increasing penalties, confirming that the failure to abstain is not due to signal degradation.
• Figure 3: Normalized Average Utility Collapses Under Risk. As penalties increase, normalized utility drops sharply into negative values on high-uncertainty benchmarks (HLE, GPQA). This confirms that models persist in answering incorrectly even when the cost of error far outweighs the potential reward.
• Figure 4: Policy Consistency Collapses Under High Penalties. We measure how often model decisions align with the optimal policy induced by their confidence. The sharp drop on HLE and GPQA shows that models fail to adjust their decision thresholds $\tau(\lambda)$ as penalties rise, persisting in answering when abstention is optimal.
• Figure 5: Internal Uncertainty Estimates Are Invariant to Risk. We analyze four calibration-related metrics across HLE, GPQA, and GSM8K as the penalty for wrong answers ($\lambda$) increases. (Top row) Verbalized confidence does not drop, proving models do not act conservatively by lowering confidence. (Rows 2-4) Calibration quality (AUARC, ECE, Brier) remains stable. This confirms that the failure to abstain is not caused by signal degradation or a loss of calibration under pressure; the signal exists, but the decision policy fails to use it.