Do explanations generalize across large reasoning models?

TL;DR

The paper investigates whether chain-of-thought explanations generalize across large reasoning models by formalizing cross-model generalization as cross-model consistency. It introduces four CoT elicitation modes (Empty, Default, Transfer, Ensemble) and a sentence-level ensembling strategy, showing that CoTs can increase cross-model consistency even when the explanations lead to incorrect answers. Human studies reveal that more generalizable CoTs align with user preferences, and reinforcement-learning post-training further enhances cross-model consistency, though not always transfer accuracy. The work provides a practical framework and actionable techniques for evaluating and enhancing generalizable CoTs, with implications for AI-assisted discovery, education, and safety-driven model supervision.

Abstract

Large reasoning models (LRMs) produce a textual chain of thought (CoT) in the process of solving a problem, which serves as a potentially powerful tool to understand the problem by surfacing a human-readable, natural-language explanation. However, it is unclear whether these explanations generalize, i.e. whether they capture general patterns about the underlying problem rather than patterns which are esoteric to the LRM. This is a crucial question in understanding or discovering new concepts, e.g. in AI for science. We study this generalization question by evaluating a specific notion of generalizability: whether explanations produced by one LRM induce the same behavior when given to other LRMs. We find that CoT explanations often exhibit this form of generalization (i.e. they increase consistency between LRMs) and that this increased generalization is correlated with human preference rankings and post-training with reinforcement learning. We further analyze the conditions under which explanations yield consistent answers and propose a straightforward, sentence-level ensembling strategy that improves consistency. Taken together, these results prescribe caution when using LRM explanations to yield new insights and outline a framework for characterizing LRM explanation generalization.

Figures (13)

• Figure 1: Methods for eliciting chain-of-thought (CoT) explanations and evaluating them. We evaluate explanation generation by querying LLMs for the answer to questions using CoTs in four different ways: (A) Empty CoT. No reasoning text is provided between the model’s thinking tags. (B) Default CoT. The model’s own reasoning is used. (2.1) uses greedy decoding, while (2.2) uses nucleus sampling. (C) Transfer CoT. Reasoning from one model is directly transferred to another, replacing its own. (D) Ensembled CoT. A generator–evaluator loop. Generator models produce $n=3$ candidate sentences ($\leq 15$ tokens each), forming $k$ candidates. These are scored by the evaluator, and the least surprising candidate (lowest perplexity) is appended to the growing ensembled thought. This updated context is fed back into the generators, and the process repeats until an end-of-thought or maximum token limit is reached. (E) After eliciting CoTs in these four settings, we evaluate their generalization to new LRMs. The transfer CoT and Ensembled CoT significantly improve the consistency of the answer produced between the model producing the original CoT and the model producing the final answer (results shown here on the MedCalc-Bench dataset).
• Figure 2: Average pairwise consistency across thought settings in MedCalc-Bench (above) and Instruction Induction (below). For thought variations indicating Ensemble CoT, models listed before the slash (/) serve as generators, while the model after the slash acts as the judge/evaluator.
• Figure 3: Consistency breakdown across thought variationsLeft: Proportion of consistent outputs separated into matching correct and matching incorrect conclusions. Right: Rate of consistent answers that are wrong across various thought settings.
• Figure 4: CoT Transfer Effect Analysis Distribution of transfer outcomes when CoT reasoning is used or transferred across models. Each CoT setting is evaluated by comparing model predictions with CoT versus without CoT (empty baseline). Settings include, Default: models using their own generated CoT; Sampled: CoT generated through sampling; Transfer CoT $l_{gen}$: CoT transferred from model $l_{gen}$ to all models; Ensemble CoT: combined CoT from multiple models. The five conditions represent: Wrong→Correct: cases where CoT successfully corrects errors (green); Correct→Wrong: cases where CoT misleads the model from correct to incorrect predictions (red); Correct→Correct: cases where CoT maintains correct predictions (blue); Wrong→Wrong(Match): both predictions incorrect with identical wrong answers (light gray); Wrong→Wrong(Diff): both predictions incorrect with different wrong answers (dark gray). Results are aggregated across multiple target models for each CoT setting. Settings are sorted by Wrong→Correct rate (descending).
• Figure 5: Comparing human user ratings of CoTs to LRM consistency (left) and LRM accuracy (right). Consistency and accuracy both show positive correlations with user ratings, with consistency showing a stronger trend. User ratings were collected for four criteria: Clarity of Steps, Ease of Following, Confidence, and Best Overall.