Through the Valley: Path to Effective Long CoT Training for Small Language Models

TL;DR

Small language models exhibit Long CoT Degradation when trained with limited long CoT supervision, showing major performance drops and longer, error-prone traces. The authors conduct a broad empirical study across Qwen2.5, LLaMA3, and Gemma3 families on OpenR1-Math-$220k$, spanning data scales from $8k$ to $220k$, and analyze mechanisms via reflection behavior and a synthetic arithmetic benchmark. They find degradation at $8k$–$16k$, but recovery improves with more data and larger models; token efficiency increases with scale, and longer CoT traces can both aid and hinder reasoning depending on data. On RL, limited long CoT SFT can impair downstream training, but sufficiently scaled long CoT SFT plus RL yields substantial gains in accuracy and token efficiency, suggesting a principled SFT+RL pipeline for small reasoning-capable systems. These findings caution against naive long CoT usage in SLMs and offer practical guidance for building more capable, efficient small-scale reasoners.

Abstract

Long chain-of-thought (CoT) supervision has become a common strategy to enhance reasoning in language models. While effective for large models, we identify a phenomenon we call Long CoT Degradation, in which small language models (SLMs; <=3B parameters) trained on limited long CoT data experience significant performance deterioration. Through extensive experiments on the Qwen2.5, LLaMA3 and Gemma3 families, we demonstrate that this degradation is widespread across SLMs. In some settings, models trained on only 8k long CoT examples lose up to 75% of their original performance before fine-tuning. Strikingly, we further observe that for some particularly small models, even training on 220k long CoT examples fails to recover or surpass their original performance prior to fine-tuning. Our analysis attributes this effect to error accumulation: while longer responses increase the capacity for multi-step reasoning, they also amplify the risk of compounding mistakes. Furthermore, we find that Long CoT Degradation may negatively impacts downstream reinforcement learning (RL), although this can be alleviated by sufficiently scaled supervised fine-tuning (SFT). Our findings challenge common assumptions about the benefits of long CoT training for SLMs and offer practical guidance for building more effective small-scale reasoning models.

Figures (15)

• Figure 1: Accuracy and response length for Qwen2.5-0.5B across varying amounts of long CoT SFT data. Performance drops markedly at smaller data scales (8k-16k), even as response length increases significantly, indicating a critical failure mode in which the model generates longer but less accurate reasoning traces. We term this phenomenon Long CoT Degradation.
• Figure 2: Comprehensive evaluation of multiple models trained with varying amounts of long CoT data. Accuracy is averaged across AIME24, AMC23, and MATH500, while response length is measured as the mean token count from 4,000 responses to MATH500. Per-benchmark results are provided in Appendix \ref{['appendix:subsec:long_cot_degradation']}.
• Figure 3: Reflection ratios of Qwen models of different sizes trained on varying amounts of long CoT data. The reflection ratio refers to the proportion of model responses (out of 4,000 on the MATH500 benchmark) that exhibit reflective behavior, as identified through cross-validation.
• Figure 4: Average response lengths of Qwen2.5-{1.5B, 3B}-Instruct models trained with varying amounts of long CoT data. Solid lines represent responses exhibiting reflection behavior; dashed lines denote responses without reflection. Results for more models are provided in Appendix \ref{['appendix:subsec:reflection_analysis']}.
• Figure 5: Arithmetic accuracy and response length on our synthetic benchmark for models trained with increasing amounts of long CoT data. Most models exhibit a sharp drop in arithmetic accuracy and a corresponding increase in response length after training on the 8k subset, with the exception of Llama-3.2-{1B,3B}-Instruct, whose initial performance is already low ($<$20% accuracy).