Benchmarking Uncertainty Calibration in Large Language Model Long-Form Question Answering

TL;DR

This work addresses the pressing need for reliable uncertainty quantification in LLM-driven scientific question answering by introducing the first large-scale calibration benchmark and an extensible open-source framework. It systematically evaluates token-level confidences, verbalized approaches, and semantic-consistency signals across up to 20 open-weight models and 7 datasets, totaling approximately 685k long-form responses. Key findings show that instruction tuning induces strong token-level probability polarization, degrading the reliability of token confidences, while sequence-level methods vary in usefulness, with Frequency of Answer offering the most robust calibration despite computational costs; reliance on ECE alone is misleading. The study provides practical benchmarks and insights that highlight critical limitations of current UQ methods in LLMs and outlines a path toward more robust, scalable calibration tools for long-form scientific QA.

Abstract

Large Language Models (LLMs) are commonly used in Question Answering (QA) settings, increasingly in the natural sciences if not science at large. Reliable Uncertainty Quantification (UQ) is critical for the trustworthy uptake of generated answers. Existing UQ approaches remain weakly validated in scientific QA, a domain relying on fact-retrieval and reasoning capabilities. We introduce the first large-scale benchmark for evaluating UQ metrics in reasoning-demanding QA studying calibration of UQ methods, providing an extensible open-source framework to reproducibly assess calibration. Our study spans up to 20 large language models of base, instruction-tuned and reasoning variants. Our analysis covers seven scientific QA datasets, including both multiple-choice and arithmetic question answering tasks, using prompting to emulate an open question answering setting. We evaluate and compare methods representative of prominent approaches on a total of 685,000 long-form responses, spanning different reasoning complexities representative of domain-specific tasks. At the token level, we find that instruction tuning induces strong probability mass polarization, reducing the reliability of token-level confidences as estimates of uncertainty. Models further fine-tuned for reasoning are exposed to the same effect, but the reasoning process appears to mitigate it depending on the provider. At the sequence level, we show that verbalized approaches are systematically biased and poorly correlated with correctness, while answer frequency (consistency across samples) yields the most reliable calibration. In the wake of our analysis, we study and report the misleading effect of relying exclusively on ECE as a sole measure for judging performance of UQ methods on benchmark datasets. Our findings expose critical limitations of current UQ methods for LLMs and standard practices in benchmarking thereof.

Figures (17)

• Figure 1: Calibration Plots Using Label Probabilities as Uncertainty Scores for Only the Most Probable Label Per Prompt. Columns correspond to three selected model families: base variants are shown in orange, instruction-tuned variants in blue, and reasoning variants in green. Rows refer to three QA datasets on MMLU, GSM8KMC and GPQA. Darker shading indicates a higher number of items within each confidence bin. Each bin is labelled with the sample count contained. The plot shows the results using Prompt Design 1 with structured decoding enabled.
• Figure 2: ECE as a Misleading Proxy for Calibration: An Illustrative Case. Two calibration plots from our results illustrate that similar or even lower ECE values do not necessarily indicate better calibration. In the left plot, uncertainty scores exhibit a clear relationship with correctness. In the right plot, there is no meaningful relationship between scores and correctness, yet the model still obtains a lower ECE and only a slightly lower AUROC.
• Figure 3: Selected Calibration Plots For The Four Selected Methods. Results for Llama-3.3-70B (left) and gpt-oss-20b (right) are shown, each for MMLU and GPQA for the four computed sequence level uncertainty methods. The full plots for all models and datasets per UQ method can be found in \ref{['sec:extensive-calibration-plots-exp2']}.
• Figure 4: Distribution of Confidence Scores Across Selected Representative Models. Confidence scores have been aggregated across all datasets ($57,500$ prompts in total). See \ref{['sec:verbalized-uncertainty-distribution']} and \ref{['sec:ptrue-distribution']} for extensive plots for all models.
• Figure A.1: Distribution of Reasoning Tokens Produced by Models. The figure displays a grouped boxplot illustrating the distribution of the Amount of Reasoning Tokens generated by different Reasoning Models across the individual Datasets. Models are grouped along the X-axis, with each boxplot representing the token count distribution for a specific dataset within that model. The central line in each box represents the median, and the box edges represent the first and third quartiles.