Your Pre-trained LLM is Secretly an Unsupervised Confidence Calibrator

TL;DR

This work tackles calibration of post-trained language models (PoLMs) using unlabeled data. It identifies that naive confidence alignment with well-calibrated pre-trained language models (PLMs) is hindered by prediction disagreement, which can drive temperature scaling $\tau$ to misleading values. The authors propose Disagreement-Aware Confidence Alignment (DACA), which restricts the temperature optimization to agreement examples, thereby producing more reliable calibration without labeled data. Empirically, DACA delivers significant ECE reductions across open-source and API-based LLMs on benchmarks like MMLU, MedMCQA, and TruthfulQA, often rivaling supervised temperature scaling while remaining unsupervised. The method is versatile, applicable with vector/matrix scaling extensions, and beneficial for selective classification, with manageable computational overhead and broad practical applicability.

Abstract

Post-training of large language models is essential for adapting pre-trained language models (PLMs) to align with human preferences and downstream tasks. While PLMs typically exhibit well-calibrated confidence, post-trained language models (PoLMs) often suffer from over-confidence, assigning high confidence to both correct and incorrect outputs, which can undermine reliability in critical applications. A major obstacle in calibrating PoLMs is the scarcity of labeled data for individual downstream tasks. To address this, we propose Disagreement-Aware Confidence Alignment (DACA), a novel unsupervised method to optimize the parameters (e.g., temperature $τ$) in post-hoc confidence calibration. Our method is motivated by the under-confidence issue caused by prediction disagreement between the PLM and PoLM while aligning their confidence via temperature scaling. Theoretically, the PLM's confidence underestimates PoLM's prediction accuracy on disagreement examples, causing a larger $τ$ and producing under-confident predictions. DACA mitigates this by selectively using only agreement examples for calibration, effectively decoupling the influence of disagreement. In this manner, our method avoids an overly large $τ$ in temperature scaling caused by disagreement examples, improving calibration performance. Extensive experiments demonstrate the effectiveness of our method, improving the average ECE of open-sourced and API-based LLMs (e.g. GPT-4o) by up to 15.08$\%$ on common benchmarks.

Figures (11)

• Figure 1: Llama-3-8B
• Figure 2: Qwen-2.5-7B
• Figure 3: DeepSeek-V2-Lite
• Figure 4: Yi-1.5-6B
• Figure 6: Under-confidence issue of naive confidence alignment. (a): Reliability diagram for Yi-1.5-9B-Chat on the computer security and college chemistry subjects of MMLU mmlu. Results of more models are presented in Appendix \ref{['section:Detail']}. (b): Temperature values of Yi-1.5-9B-Chat under different training epochs when trained separately on the disagreement and agreement sets and the whole dataset. The training process is performed on the computer security and the college chemistry subject of MMLU.

Theorems & Definitions (5)

• Definition 3.1: Unlabeled data distribution
• Proposition 3.2
• Proposition 3.3
• proof
• proof