Distilling the Essence: Efficient Reasoning Distillation via Sequence Truncation

TL;DR

This work tackles the cost of distilling reasoning from large language models by analyzing which parts of a reasoning sequence (prompt, chain-of-thought, and answer) are essential for transfer to a smaller student. It introduces a Lead-Span Proportion truncation protocol to quantify computation–quality tradeoffs and conducts controlled experiments with Qwen3 teacher–student pairs on math reasoning benchmarks. The key finding is that supervising the chain-of-thought is the dominant source of transferable reasoning, and training on the first $50\%$ of tokens retains about $94\%$ of full-sequence performance while cutting training time, memory, and FLOPs by around $50\%$, making reasoning distillation far more scalable. This suggests a practical, simple lever—early-token prioritization—for deploying efficient, capable reasoning models.

Abstract

Distilling the reasoning capabilities from a large language model (LLM) to a smaller student model often involves training on substantial amounts of reasoning data. However, distillation over lengthy sequences with prompt (P), chain-of-thought (CoT), and answer (A) segments makes the process computationally expensive. In this work, we investigate how the allocation of supervision across different segments (P, CoT, A) affects student performance. Our analysis shows that selective knowledge distillation over only the CoT tokens can be effective when the prompt and answer information is encompassed by it. Building on this insight, we establish a truncation protocol to quantify computation-quality tradeoffs as a function of sequence length. We observe that training on only the first $50\%$ of tokens of every training sequence can retain, on average, $\approx94\%$ of full-sequence performance on math benchmarks while reducing training time, memory usage, and FLOPs by about $50\%$ each. These findings suggest that reasoning distillation benefits from prioritizing early reasoning tokens and provides a simple lever for computation-quality tradeoffs. Codes are available at https://github.com/weiruichen01/distilling-the-essence.

Figures (15)

• Figure 1: Accuracy on AIME24 and AIME25 under different supervision regions for two different training datasets. Supervising CoT tokens is crucial, with setups that ignore CoT scoring lowest and only modest differences among CoT-inclusive variants.
• Figure 2: A training example. The text is continuous but divided into three sections to illustrate the idea of Prompt (P), CoT traces (CoT), and Answer (A). We observe that the information in P and A is often covered within CoT. Some intermediate text in CoT has been omitted here for readability. The full example is provided as Figure \ref{['fig:complete_training_ex_Abe']} in Appendix \ref{['appendix:train_dataset_info']}.
• Figure 3: Sequence-length Scaling behavior of accuracy against different Lead-Span Proportions (LSP) shows that performance is not linear. The vast majority of performance can be retained when training with only a portion of tokens for each training example. Since student models smaller than 4B performed near baseline on OpenThoughts, we restricted our experiments on Bespoke to student models larger than or equal to 4B.
• Figure 4: Impact of $50\%$ budget allocation strategies (left, right) on model accuracy. Retaining the left portion of the sequence consistently outperforms the right, indicating that early tokens are more valuable for learning than the later tokens. This ablation of training budget verifies that the efficiency of using 50% tokens is not a general trait when reducing the number of training tokens.
• Figure 5: First half of tokens tends to contain denser useful information than the second half, in line with stronger performance when training on the left $50\%$ of tokens. Two complete examples are illustrated in Figure \ref{['fig:complete_training_ex_Abe']} and Figure \ref{['fig:complete_training_ex_caterpillar']} in Appendix \ref{['appendix:train_dataset_info']}.