Long-Short Chain-of-Thought Mixture Supervised Fine-Tuning Eliciting Efficient Reasoning in Large Language Models

TL;DR

<3-5 sentence high-level summary> The paper addresses how to transfer reasoning abilities from large reasoning models to non-reasoning LLMs without inheriting the overthinking and verbose chains of thought. It introduces LS-Mixture SFT, a three-stage approach that rewrites longCoT into shortCoT with structure preservation, trains on a mixture of long and short CoT data, and uses an inference-time balanced thinking mode. Experiments on MATH500, AIME24, and GPQA show improved accuracy (up to +6.7%) and substantial reductions in response length (≈47.6%), while also reducing training sequence length. Ablations confirm the importance of structure-preserved rewriting and long-short data mixing, and the method is validated in open-source datasets/models (s1K-mix, s1-mix-32B).

Abstract

Recent advances in large language models have demonstrated that Supervised Fine-Tuning (SFT) with Chain-of-Thought (CoT) reasoning data distilled from large reasoning models (e.g., DeepSeek R1) can effectively transfer reasoning capabilities to non-reasoning models. However, models fine-tuned with this approach inherit the "overthinking" problem from teacher models, producing verbose and redundant reasoning chains during inference. To address this challenge, we propose Long-Short Chain-of-Thought Mixture Supervised Fine-Tuning (LS-Mixture SFT), which combines long CoT reasoning dataset with their short counterparts obtained through structure-preserved rewriting. Our experiments demonstrate that models trained using the LS-Mixture SFT method, compared to those trained with direct SFT, achieved an average accuracy improvement of 2.3% across various benchmarks while substantially reducing model response length by approximately 47.61%. This work offers an approach to endow non-reasoning models with reasoning capabilities through supervised fine-tuning while avoiding the inherent overthinking problems inherited from teacher models, thereby enabling efficient reasoning in the fine-tuned models.

Figures (5)

• Figure 1: Overview of LS-Mixture SFT. This method consists of three stages: 1) Structure-preserved CoT Rewriting: A LLM is used to rewrite the long CoT trajectories into short ones while preserving the core structure. 2) Mixture Supervised Fine-Tuning: Non-reasoning LLM is been supervised fine-tuned on mixture datasets. 3) Inference-time Balanced Thinking: The fine-tuned model is designed to employ a balanced thinking mode that lies between detailed and brief when generating reasoning responses to queries.
• Figure 2: Comparison of response length distribution between s1.1-32B and s1-mix-32B models on the MATH500 evaluation task. The horizontal axis represents response length ranges (in string length), while the vertical axis shows the number of samples within each length range.
• Figure 3: Comparison of mean response lengths for correct (red) and error (green) predictions of the s1-mix-32B model across the MATH500, GPQA, and AIME24 evaluation datasets. The figure also presents a weighted average of these results, with weights determined by the size of each respective dataset.
• Figure 4: Word cloud of CoT trajectories in the long reasoning dataset.
• Figure 5: Word cloud of CoT trajectories in the short reasoning dataset.