Trained on Tokens, Calibrated on Concepts: The Emergence of Semantic Calibration in LLMs
Preetum Nakkiran, Arwen Bradley, Adam Goliński, Eugene Ndiaye, Michael Kirchhof, Sinead Williamson
TL;DR
This work investigates the emergence of semantic calibration in large language models (LLMs) byIntroducing a formal framework based on semantic collapsing functions $B$ and the induced category distributions $\pi_x = B_x \sharp p_x$. The authors prove that $B$-confidence-calibration is equivalent to local loss optimality with respect to a class of semantic perturbations $\mathcal{W}_B$, suggesting calibration arises as a byproduct of next-token prediction if the model can anticipate its own semantic distribution early in generation. They further show that perturbations are efficiently implementable via simple circuits when the model has access to intermediate $B$-confidences, leading to testable predictions: base LLMs should exhibit semantic calibration across open-domain QA; RL instruction-tuning and chain-of-thought can degrade calibration. Through extensive experiments on GSM8K, OpenMathInstruct-2, TriviaQA, and SimpleQA with models from 0.5B to 72B, they demonstrate that base models are indeed semantically calibrated in non-CoT settings, while instruction-tuned or CoT configurations often break calibration, with a measurable correlation between the learnability of $B_x \sharp p_x$ (via LoRA probes) and calibration performance. The results provide a principled explanation for when and why semantic calibration emerges, offering insights into LLM uncertainty and design implications for training regimes and prompting strategies. The work highlights limitations around the specific calibration notion studied and calls for future work on broader calibration forms and practical deployment considerations.
Abstract
Large Language Models (LLMs) often lack meaningful confidence estimates for their outputs. While base LLMs are known to exhibit next-token calibration, it remains unclear whether they can assess confidence in the actual meaning of their responses beyond the token level. We find that, when using a certain sampling-based notion of semantic calibration, base LLMs are remarkably well-calibrated: they can meaningfully assess confidence in open-domain question-answering tasks, despite not being explicitly trained to do so. Our main theoretical contribution establishes a mechanism for why semantic calibration emerges as a byproduct of next-token prediction, leveraging a recent connection between calibration and local loss optimality. The theory relies on a general definition of "B-calibration," which is a notion of calibration parameterized by a choice of equivalence classes (semantic or otherwise). This theoretical mechanism leads to a testable prediction: base LLMs will be semantically calibrated when they can easily predict their own distribution over semantic answer classes before generating a response. We state three implications of this prediction, which we validate through experiments: (1) Base LLMs are semantically calibrated across question-answering tasks, (2) RL instruction-tuning systematically breaks this calibration, and (3) chain-of-thought reasoning breaks calibration. To our knowledge, our work provides the first principled explanation of when and why semantic calibration emerges in LLMs.