Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Merge-of-Thought Distillation

Zhanming Shen, Zeyu Qin, Zenan Huang, Hao Chen, Jiaqi Hu, Yihong Zhuang, Guoshan Lu, Gang Chen, Junbo Zhao

TL;DR

This work tackles distilling long chain-of-thought (CoT) reasoning from numerous teacher LLMs into a compact student. It introduces Merge-of-Thought Distillation (MoT), an iterative framework that alternates teacher-specific supervised fine-tuning branches with weight-space merging to fuse diverse reasoning signals into a consensus student. Across BOBA-200 and S1K-200 with roughly 200 CoT examples, MoT outperforms single-teacher distillation and naive multi-teacher unions, improving general reasoning while mitigating forgetting and showing robustness to distribution shifts and peer-level teachers. The approach also reveals that consensus CoT emerges naturally under MoT, evidenced by token-level analyses and loss-landscape flattening, enabling effective transfer to new student models and selection-free CoT distillation.

Abstract

Efficient reasoning distillation for long chain-of-thought (CoT) models is increasingly constrained by the assumption of a single oracle teacher, despite the practical availability of multiple candidate teachers and growing CoT corpora. We revisit teacher selection and observe that different students have different "best teachers," and even for the same student, the best teacher can vary across datasets. Therefore, to unify multiple teachers' reasoning abilities into a student to overcome conflicts among various teachers' supervision, we propose Merge-of-Thought Distillation (MoT), a lightweight framework that alternates between teacher-specific supervised fine-tuning branches and weight-space merging of the resulting student variants. On competition math benchmarks, using only about 200 CoT samples, applying MoT to a Qwen3-14B student surpasses strong models including Deepseek-R1, Qwen3-32B, and OpenAI-O1, demonstrating substantial gains. Besides, MoT consistently outperforms the best single-teacher distillation, improves general reasoning beyond mathematics while reducing catastrophic forgetting, and shows robustness to distribution-shifted and peer-level teachers. Finally, we have demonstrated MoT possesses consensus CoT by eliminating teacher-specific inductive biases and inter-teacher conflicts while repeatedly reinforcing the learning of consensus reasoning features. These results position MoT as a simple, effective route to efficiently distilling long CoT capabilities from diverse teachers into compact students.

Merge-of-Thought Distillation

TL;DR

This work tackles distilling long chain-of-thought (CoT) reasoning from numerous teacher LLMs into a compact student. It introduces Merge-of-Thought Distillation (MoT), an iterative framework that alternates teacher-specific supervised fine-tuning branches with weight-space merging to fuse diverse reasoning signals into a consensus student. Across BOBA-200 and S1K-200 with roughly 200 CoT examples, MoT outperforms single-teacher distillation and naive multi-teacher unions, improving general reasoning while mitigating forgetting and showing robustness to distribution shifts and peer-level teachers. The approach also reveals that consensus CoT emerges naturally under MoT, evidenced by token-level analyses and loss-landscape flattening, enabling effective transfer to new student models and selection-free CoT distillation.

Abstract

Efficient reasoning distillation for long chain-of-thought (CoT) models is increasingly constrained by the assumption of a single oracle teacher, despite the practical availability of multiple candidate teachers and growing CoT corpora. We revisit teacher selection and observe that different students have different "best teachers," and even for the same student, the best teacher can vary across datasets. Therefore, to unify multiple teachers' reasoning abilities into a student to overcome conflicts among various teachers' supervision, we propose Merge-of-Thought Distillation (MoT), a lightweight framework that alternates between teacher-specific supervised fine-tuning branches and weight-space merging of the resulting student variants. On competition math benchmarks, using only about 200 CoT samples, applying MoT to a Qwen3-14B student surpasses strong models including Deepseek-R1, Qwen3-32B, and OpenAI-O1, demonstrating substantial gains. Besides, MoT consistently outperforms the best single-teacher distillation, improves general reasoning beyond mathematics while reducing catastrophic forgetting, and shows robustness to distribution-shifted and peer-level teachers. Finally, we have demonstrated MoT possesses consensus CoT by eliminating teacher-specific inductive biases and inter-teacher conflicts while repeatedly reinforcing the learning of consensus reasoning features. These results position MoT as a simple, effective route to efficiently distilling long CoT capabilities from diverse teachers into compact students.

Paper Structure

This paper contains 45 sections, 1 theorem, 19 equations, 6 figures, 15 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Long Chain-of-Thought Distillation.
  4. Model Merging in LLMs.
  5. Revisiting Multi-Teacher Long CoT Distillation
  6. Setup and goals.
  7. Different students have different best teachers.
  8. Simple mixing or one-shot post-hoc merging is insufficient.
  9. Method: Merge-of-Thought Distillation (MoT)
  10. Initialization
  11. Teacher-Specific SFT (Branch Training)
  12. Weight-Space Merging and Iteration
  13. Experiments Setup
  14. Multi-teacher distillation and MoT yield substantial gains
  15. Performance on competition math benchmarks
  16. ...and 30 more sections

Key Result

Lemma 1

For any fixed $E\ge1$ and $\eta>0$, $s_E(\lambda)=\eta\sum_{e=0}^{E-1}(1-\eta\lambda)^e$ is strictly decreasing in $\lambda$ on $(0,2/\eta)$.

Figures (6)

  • Figure 1: Teacher choice is not universal. Left: different students have different "best teachers"; right: even for the same student the best teacher can vary across datasets. This observation is empirically confirmed in Table \ref{['tab:best-teacher-by-base']}.
  • Figure 2: Workflow of Merge-of-Thought Distillation (MoT). After the candidate teachers generate the teacher-specific distillation dataset based on the seed problem, the system enters the iterative MoT algorithm process. In each round $t$, we perform three steps: Step 1 (branch training): initialize $K$ branches from the current merged student and train each on its teacher-specific distillation dataset $\mathcal{D}^{(k)}$ (Eq. \ref{['eq:sft']}); Step 2 (weight merge): average the branch parameters in weight space to obtain the aggregated model $\theta^{(t)}$ (Eq. \ref{['eq:merge']}); Step 3 (next-round initialization): use $\theta^{(t)}$ as the base initialization for round $t{+}1$.
  • Figure 3: Qwen3-8B under MoT vs. STD (QWQ) on S1K and BOBA. Panels (a,b): S1K; panels (c,d): BOBA. Left columns show training loss vs. steps; right columns show AIME vs. steps. All runs log loss at every step on the same QWQ-distilled corpus; AIME is evaluated every 50 steps.
  • Figure 4: BOBA dataset: MoT vs. MoT without R1 at two scales. Panels (a,b): 8B; panels (c,d): 14B. Left columns show training loss vs. steps; right columns show AIME vs. steps. All runs log loss at every step on the same QWQ-distilled corpus; AIME is evaluated every 50 steps.
  • Figure 5: Tokens marked with confidence drops relative to the Base model after MoT.
  • ...and 1 more figures

Theorems & Definitions (3)

  • Lemma 1: Monotonicity of $s_E$
  • Remark 1: Implicit proximal effect (Mitigating Forgetting)
  • Remark 2: Unified Improvements (Mitigating Conflicts)