Calibrating Reasoning in Language Models with Internal Consistency

TL;DR

This work tackles the reliability of reasoning in large language models by focusing on internal representations during chain-of-thought reasoning. It introduces internal consistency as a training-free, self-evaluative metric that measures agreement between latent predictions from intermediate layers and final outputs. By weighting reasoning paths by their internal-consistency scores, the authors achieve consistent improvements in calibrated accuracy across reading comprehension, symbolic, and logical reasoning tasks, and show that layer-specific aggregation patterns generalize across datasets. The study also analyzes how transformer components contribute to internal inconsistency, offering mechanistic insights and practical calibration methods while highlighting limitations and potential societal implications.

Abstract

Large language models (LLMs) have demonstrated impressive capabilities in various reasoning tasks, aided by techniques like chain-of-thought prompting that elicits verbalized reasoning. However, LLMs often generate text with obvious mistakes and contradictions, raising doubts about their ability to robustly process and utilize generated rationales. In this work, we investigate reasoning in LLMs through the lens of internal representations, focusing on how these representations are influenced by generated rationales. Our preliminary analysis reveals that while generated rationales improve answer accuracy, inconsistencies emerge between the model's internal representations in middle layers and those in final layers, potentially undermining the reliability of their reasoning processes. To address this, we propose internal consistency as a measure of the model's confidence by examining the agreement of latent predictions decoded from intermediate layers. Extensive empirical studies across different models and datasets demonstrate that internal consistency effectively distinguishes between correct and incorrect reasoning paths. Motivated by this, we propose a new approach to calibrate reasoning by up-weighting reasoning paths with high internal consistency, resulting in a significant boost in reasoning performance. Further analysis uncovers distinct patterns in attention and feed-forward modules across layers, providing insights into the emergence of internal inconsistency. In summary, our results demonstrate the potential of using internal representations for self-evaluation of LLMs. Our code is available at github.com/zhxieml/internal-consistency.

Figures (10)

• Figure 1: An illustration of internal consistency. Given a true-or-false question with the ground truth being true, we elicit latent predictions (i.e., predictions decoded from intermediate layers, represented by "T" for true and "F" for false) from the answer token of reasoning paths. By defining internal consistency as the agreement of latent predictions with the final stated one, we observe a high correlation between internal consistency and prediction accuracy, which aids in calibrating reasoning. Note that the figure on the right is synthesized for illustration purposes, but the distributions reflect actual observations as shown in Figure \ref{['fig:boxplot_hist']}.
• Figure 2: CoT reasoning improves answer accuracy but exacerbates the inconsistency between hidden and stated reasoning. Left: Heatmap of linear probe accuracies at all reasoning steps and intermediate layers. Right: A zoom-in on the results of the last two steps.
• Figure 3: Internal consistency is a reliable measure of prediction confidence in CoT reasoning. From left to right: 1) the effect of different prompting techniques on the model's internal consistency; 2) the distribution discrepancy of internal consistency between correct and incorrect model predictions; 3) pattern variations in agreement values (representing the ratio of data instances where the latent predictions match the final predictions) across layers; and 4) a calibration plot with bins according to the model’s internal consistency on the x-axis and the accuracy within each bin on the y-axis. Results are averaged over all models and datasets. See Appendix \ref{['appendix:more']} for the full results.
• Figure 4: Internal consistency brings larger gains for complex reasoning tasks. The figure shows calibrated accuracy as a function of the number of votes for three types of tasks: reading comprehension (BoolQ), symbolic reasoning (CoinFlip), and logical reasoning (PrOntoQA and ProofWriter). Results are averaged over all models and random seeds.
• Figure 5: The emergence of internal inconsistency in CoT reasoning could be attributed to the misalignment between layers with high attention weights on critical tokens and those promotes certain predictions. The histograms display attention weights for each part (context, query, and rationale) across self-attention layers , accompanied by a gray line indicating the count of value vectors in FFN layers that achieve high cosine similarity to the model's final prediction.