Probabilities of Chat LLMs Are Miscalibrated but Still Predict Correctness on Multiple-Choice Q&A

TL;DR

This study assesses whether the MSP of chat-tuned LLMs can reliably indicate correctness on MCQ tasks. Although MSPs are systematically miscalibrated, they retain a strong, increasing correlation with actual correctness as model QA performance improves, enabling out-of-the-box correctness prediction and a practical abstention mechanism with minimal labeled data. Unlike calibration, which does not naturally improve with model capability, MSP-based correctness signals become sharper with more capable models. The work also demonstrates a proof-of-concept abstention strategy that improves final scores, highlighting a usable uncertainty-aware approach for reducing erroneous answers in real-world Q&A deployments.

Abstract

We study 15 large language models (LLMs) fine-tuned for chat and find that their maximum softmax probabilities (MSPs) are consistently miscalibrated on multiple-choice Q&A. However, those MSPs might still encode useful uncertainty information. Specifically, we hypothesized that wrong answers would be associated with smaller MSPs compared to correct answers. Via rigorous statistical testing, we show that this hypothesis holds for models which perform well on the underlying Q&A task. We also find a strong direction correlation between Q&A accuracy and MSP correctness prediction, while finding no correlation between Q&A accuracy and calibration error. This suggests that within the current fine-tuning paradigm, we can expect correctness prediction but not calibration to improve as LLM capabilities progress. To demonstrate the utility of correctness prediction, we show that when models have the option to abstain, performance can be improved by selectively abstaining based on the MSP of the initial model response, using only a small amount of labeled data to choose the MSP threshold.

Figures (33)

• Figure 1: A sample question prompt.
• Figure 2: Left: The calibration curve for each model. Most models exhibit clear overconfidence: the MSP is larger than the true fraction of correct responses. Right: The average calibration error per model (scaled by 100 to improve readability) vs Q&A accuracy. There is no statistical evidence for a correlation between calibration error and Q&A accuracy ($R^2 = 0.07, p = 0.33$). The precise calibration error values can be found in the appendix (\ref{['tab:calibration_prob_max']}).
• Figure 2: Main AUROC results. AUROC and Q&A values are percentages, averaged over ten data points (five datasets and two phrasings). The $p < 10^{-4}$ columns indicate how many of those ten data points yielded $p$-values below $10^{-4}$ for the null hypothesis that AUROC $= 50\%$. The $p$-values are from the Mann-Whitney U test; see Section \ref{['sec:method']} for details.
• Figure 4: AUROC vs model size for MSP (left) and Max Logit (right). The coefficients of determination for MSP and Max Logit were $R^2=0.35$ ($p = 0.03$) and $R^2=0.16$ ($p=0.17$) respectively. GPT-3.5 Turbo and GPT-4o were excluded since their sizes are unknown.
• Figure 5: Average AUROC vs Q&A accuracy based on prompt phrasing (the two phrasings can be found in Figures \ref{['fig:first-prompt']} and \ref{['fig:second-prompt']}). All values are averaged over the five datasets.