Calibration Is Not Enough: Evaluating Confidence Estimation Under Language Variations

TL;DR

The paper argues that traditional CE evaluation for LLMs—primarily focusing on calibration and discrimination—fails to capture how confidence judgments behave under language variation. It introduces three metrics—robustness to prompt perturbations (P-RB), stability across semantically equivalent answers (A-STB), and sensitivity to semantically different answers (A-SST)—to provide a more complete assessment of CE reliability. Through extensive experiments on nine LLMs across four QA datasets and five CE methods, the study demonstrates that good calibration or AUROC does not guarantee robustness or meaningful discrimination between semantically distinct answers. The findings guide practitioners in selecting CE methods based on application priorities (robustness and stability vs. sensitivity) and call for CE evaluations that account for language variation in real-world deployments. Overall, the framework highlights limitations of existing CE evaluations and offers actionable guidance for designing more reliable CE in LLM systems.

Abstract

Confidence estimation (CE) indicates how reliable the answers of large language models (LLMs) are, and can impact user trust and decision-making. Existing work evaluates CE methods almost exclusively through calibration, examining whether stated confidence aligns with accuracy, or discrimination, whether confidence is ranked higher for correct predictions than incorrect ones. However, these facets ignore pitfalls of CE in the context of LLMs and language variation: confidence estimates should remain consistent under semantically equivalent prompt or answer variations, and should change when the answer meaning differs. Therefore, we present a comprehensive evaluation framework for CE that measures their confidence quality on three new aspects: robustness of confidence against prompt perturbations, stability across semantic equivalent answers, and sensitivity to semantically different answers. In our work, we demonstrate that common CE methods for LLMs often fail on these metrics: methods that achieve good performance on calibration or discrimination are not robust to prompt variations or are not sensitive to answer changes. Overall, our framework reveals limitations of existing CE evaluations relevant for real-world LLM use cases and provides practical guidance for selecting and designing more reliable CE methods.

Figures (9)

• Figure 1: Design of the proposed evaluation metrics for LLM confidence. We assess robustness by measuring the variation in confidence scores elicited by semantically equivalent prompt perturbations. Stability measures confidence consistency across semantically equivalent answers, while sensitivity evaluates whether CE methods assign more distinct confidence scores to semantically different answers than to equivalent ones.
• Figure 2: Comparison of different evaluation metrics for confidence estimator, based on the data they require. While other metrics use correctness labels, P-RB, A-STB and A-SST solely rely on measurements from different model answers and prompts.
• Figure 3: Averaged performance ($\uparrow$) among all models. The percentage indicates the amount of data that fulfills the requirement for evaluation of the metric (more than one answer in $\hat{\mathcal{Y}}^\text{max}$ for A-STB, and at least two semantically different answer groups for A-SST).
• Figure 4: Pearson correlation of metrics.
• Figure 5: Performance of CE methods evaluated on different models.

Theorems & Definitions (3)

• Definition 4.1: Robustness, P-RB
• Definition 4.2: Stability, A-STB
• Definition 4.3: Sensitivity, A-SST