Investigating Uncertainty Calibration of Aligned Language Models under the Multiple-Choice Setting
Guande He, Peng Cui, Jianfei Chen, Wenbo Hu, Jun Zhu
TL;DR
The paper tackles miscalibration in aligned language models under multiple-choice evaluation by isolating two sources of uncertainty—answer and format—and showing that standard alignment conflates them, driving overconfidence. It presents a formal decomposition of predictive uncertainty, analyzes how alignment stages affect each component, and demonstrates that in-context learning helps pre-trained LMs calibrate via format signaling while alignment disrupts this calibration. A practical contribution is a few-shot post-hoc calibration method that uses the pre-trained LM’s predictive distribution to calibrate aligned LMs, outperforming baseline temperature scaling and KDE approaches across tasks. The findings highlight the need for alignment processes that disentangle answer and formatting behaviors, with implications for safer, more reliable deployment of aligned LMs in decision-critical settings.
Abstract
Despite the significant progress made in practical applications of aligned language models (LMs), they tend to be overconfident in output answers compared to the corresponding pre-trained LMs. In this work, we systematically evaluate the impact of the alignment process on logit-based uncertainty calibration of LMs under the multiple-choice setting. We first conduct a thoughtful empirical study on how aligned LMs differ in calibration from their pre-trained counterparts. Experimental results reveal that there are two distinct uncertainties in LMs under the multiple-choice setting, which are responsible for the answer decision and the format preference of the LMs, respectively. Then, we investigate the role of these two uncertainties on aligned LM's calibration through fine-tuning in simple synthetic alignment schemes and conclude that one reason for aligned LMs' overconfidence is the conflation of these two types of uncertainty. Furthermore, we examine the utility of common post-hoc calibration methods for aligned LMs and propose an easy-to-implement and sample-efficient method to calibrate aligned LMs. We hope our findings could provide insights into the design of more reliable alignment processes for LMs.