Calibration Across Layers: Understanding Calibration Evolution in LLMs

TL;DR

This work reveals that calibration in transformer LLMs is not confined to the final projection but unfolds across depth, with a later calibration correction phase in upper layers. By analyzing multiple open-weight models on MCQA tasks (notably MMLU), the authors identify a low‑dimensional calibration direction in the residual stream whose perturbation improves $ECE$ and $MCE$ without reducing accuracy, suggesting a distributed, controllable confidence mechanism. The results show a consistent pattern of accuracy saturating in mid-layers while calibration metrics worsen then improve in later layers, implying internal confidence regulation beyond the last layer. These insights have implications for interpretability and reliability, indicating potential for intra‑model confidence modulation and more robust downstream decision making in LLMs.

Abstract

Large Language Models (LLMs) have demonstrated inherent calibration capabilities, where predicted probabilities align well with correctness, despite prior findings that deep neural networks are often overconfident. Recent studies have linked this behavior to specific components in the final layer, such as entropy neurons and the unembedding matrix null space. In this work, we provide a complementary perspective by investigating how calibration evolves throughout the network depth. Analyzing multiple open-weight models on the MMLU benchmark, we uncover a distinct confidence correction phase in the upper/later layers, where model confidence is actively recalibrated after decision certainty has been reached. Furthermore, we identify a low-dimensional calibration direction in the residual stream whose perturbation significantly improves calibration metrics (ECE and MCE) without harming accuracy. Our findings suggest that calibration is a distributed phenomenon, shaped throughout the network forward pass, not just in its final projection, providing new insights into how confidence-regulating mechanisms operate within LLMs.

Figures (21)

• Figure 1: The figure shows performance (Accuracy) along with model calibration scores (ECE and MCE) of the phi-2 model on the MMLU Humanities dataset. We observe that the model performance remains near-random (25%, 4-options) for initial layers and starts to rise from layer 22 and saturates at layer 26, with minor changes in the 26-31 layers. However, the ECE and MCE scores first rise (layers 25-28) and then decline (layers 28-31), highlighting the model calibration changing in the later layers.
• Figure 2: The figure provides an overview of the performed study. The Residual stream signals from each of the layers are projected back to the vocabulary space using the unembedding matrix. The obtained predictions are inspected for accuracy and model calibration scores (ECE/MCE). The models show a sudden peak arising in the middle layers, after which the performance remains saturated. Interestingly, the model goes into a calibration correction phase where the ECE first rises and then reduces, while maintaining the same accuracy, i.e. going from a poorly calibrated predictions to calibrated predictions (shown as reliability diagrams, also see Figure \ref{['fig:all_layers_calibration_relialibility_diagrams']}).
• Figure 3: The figure shows performance (Accuracy) along with model calibration scores (ECE and MCE) of the Phi-2 model on the different datasets. We observe that the model performance starts to rise from layer 22 and saturates at layer 25/26, with minor changes in the 26-31 layers. However, the ECE and MCE scores first rise (layers 26-28) and then decline (layers 29-31), highlighting calibration changing in the later layers, with meager changes in the model performance. The upper/later layers show the presence of calibration correction phase. Similar trends are found for other models (Llama-3-8B Figure \ref{['fig:all_layers_calibration_Llama-3-8B_mmlu_all']}, Mistral-7B Figure \ref{['fig:all_layers_calibration_mistralai_Mistral-7B-v0.1']}, and Llama-2-7B Figure \ref{['fig:all_layers_calibration_meta-llama_Llama-2-7b-hf_mmlu_all']}).
• Figure 4: The figure shows performance (Accuracy) along with model calibration scores (ECE and MCE) of the phi-2 model computed by reconstructing the unembedding matrix, using only top-85%, top-90% and top-95% singular values in (a), (b), and (c), respectively. Overall, we observe the ECE scores with minor fluctuations, pointing towards a small contribution of lower singular values in model calibration.
• Figure 5: The figure shows performance (Accuracy) along with model calibration scores (ECE and MCE) of the phi-2 model on the different datasets when adding the calibration direction to the residual stream. The added calibration direction to the residual stream helps shift the calibration scores to lower values, validating the impact of the calibration direction. Interestingly, the direction found using MMLU Humanities works well for other datasets like MMLU Others. (Due to space constraints, we move the results on other datasets to the App. Figure \ref{['fig:all_layers_calibration_phi-2_mmlu_all_truthfulqa_intervention']})