Which Reasoning Trajectories Teach Students to Reason Better? A Simple Metric of Informative Alignment

TL;DR

The paper tackles the challenge of data–student suitability in reasoning-focused distillation, arguing that stronger teachers do not automatically yield better students. It introduces Rank-Surprisal Ratio (RSR), defined as $RSR(\boldsymbol{x}) = \frac{\sum_k \min(\mathrm{Rank}(t_k), r_{\max})}{\sum_k \mathrm{Surprisal}(t_k)}$, to jointly capture informativeness and alignment, and demonstrates that RSR correlates strongly with post-training performance across five student models and eleven teachers (average Spearman ≈ 0.86). Through extensive experiments, the authors show that RSR enables effective trajectory selection and teacher selection, often outperforming probability-based and other baselines, and achieving results close to oracle best-teacher selections in low-resource scenarios. The findings underscore informative alignment as a practical and scalable criterion for data engineering in reasoning distillation, with implications for broader SFT and educational AI contexts. Overall, RSR provides a principled, computationally efficient tool for identifying high-signal, well-aligned reasoning data for targeted student models.

Abstract

Long chain-of-thought (CoT) trajectories provide rich supervision signals for distilling reasoning from teacher to student LLMs. However, both prior work and our experiments show that trajectories from stronger teachers do not necessarily yield better students, highlighting the importance of data-student suitability in distillation. Existing methods assess suitability primarily through student likelihood, favoring trajectories that closely align with the model's current behavior but overlooking more informative ones. Addressing this, we propose Rank-Surprisal Ratio (RSR), a simple metric that captures both alignment and informativeness to assess the suitability of a reasoning trajectory. RSR is motivated by the observation that effective trajectories typically combine low absolute probability with relatively high-ranked tokens under the student model, balancing learning signal strength and behavioral alignment. Concretely, RSR is defined as the ratio of a trajectory's average token-wise rank to its average negative log-likelihood, and is straightforward to compute and interpret. Across five student models and reasoning trajectories from 11 diverse teachers, RSR strongly correlates with post-training performance (average Spearman 0.86), outperforming existing metrics. We further demonstrate its practical utility in both trajectory selection and teacher selection.

Figures (3)

• Figure 1: Illustration of the intuition behind RSR. Suitable reasoning trajectories should balance informativeness and alignment by having low absolute probability but relatively high-ranked tokens under the student model.
• Figure 2: Simulation of the student model’s token-level prediction distribution $Z$. Tokens with low probability but relatively high rank—characteristic of the $Z_B$ mode—yield smaller token-level rank–surprisal ratios.
• Figure 3: Simulation of the student model’s token-level prediction distribution $Z$, a mixture of Zipf distributions $Z_A$ and $Z_B$.