"The Whole Is Greater Than the Sum of Its Parts": A Compatibility-Aware Multi-Teacher CoT Distillation Framework

TL;DR

This work tackles the bottleneck of Chain-of-Thought distillation by enabling adaptive, instance-level fusion of multiple teacher rationales, rather than relying on a single teacher or static merges. It introduces COMPACT, which uses three metrics—Graph-based Consensus, Mutual Information Adaptability, and Loss-based Difficulty—to compute per-teacher weights $\alpha_k(x)$ and fuse teacher updates via $\theta_{new} = \theta_{old} + \sum_k \alpha_k(x) \Delta \theta_k$, with further refinements through a consistency constraint and an overall objective $\mathcal{L}_{Final} = \sum_k \alpha_k(x) \mathcal{L}_k$. Empirical results on ID and OOD benchmarks demonstrate state-of-the-art performance for lightweight student models, along with latent-space analyses showing preserved representational structure and reduced catastrophic forgetting. The work also provides extensive ablations, confirming the necessity of each component (consensus, information gain, and difficulty) for robust, generalized reasoning transfer. Overall, COMPACT offers a data-efficient, scalable pathway to distill diverse reasoning capabilities into compact models without eroding their foundational knowledge.

Abstract

Chain-of-Thought (CoT) reasoning empowers Large Language Models (LLMs) with remarkable capabilities but typically requires prohibitive parameter scales. CoT distillation has emerged as a promising paradigm to transfer reasoning prowess into compact Student Models (SLMs), but existing approaches often rely on a solitary teacher, capping the student's potential since individual LLMs often exhibit distinct capability biases and may suffer from catastrophic forgetting. While leveraging diverse teachers seems appealing, effectively fusing their supervisions remains challenging: teacher-student incompatibility risks amplifying hallucinations, and passive supervision fails to ensure genuine logic internalization. To address this, we introduce COMPACT, a framework that adaptively fuses supervisions from different teachers by dynamically weighting teacher gradients based on the student's real-time compatibility evaluated by a multi-dimensional metric: (1) Graph-based Consensus to filter misleading rationales by identifying mainstream reasoning paths; (2) Mutual-Information-based Adaptability to detect "epiphany moments" for genuinely understanding the reasoning process rather than merely imitating; and (3) Loss-based Difficulty to assess student receptivity to the teacher's guidance and prevent negative transfer. Extensive experiments and latent space analysis demonstrate that COMPACT effectively integrates diverse reasoning capabilities without damaging the model's original knowledge structure, achieving state-of-the-art performance on various benchmarks while mitigating catastrophic forgetting.

Figures (9)

• Figure 1: Overview of our COMPACT framework. The framework consists of three main components: (1) A frozen Teacher Model Pool generating diverse rationales; (2) A Multi-Dimensional Evaluation Metric that dynamically computes Adaptability ($S_{MI}$), Consensus ($S_{cons}$), and Difficulty ($S_{PPL}$) scores; and (3) An Adaptive Gradient Fusion mechanism that updates the student model using compatibility-aware dynamic weights $\alpha_k(x)$ derived from these scores.
• Figure 2: Fine-grained capability evaluation on FLASK framework. (a) compares teachers (DeepSeek-R1-Distill-Llama-70B, QWQ-32B, Qwen2.5-72B, Llama3.1-70B)(solid lines) and their corresponding distilled students (dashed lines), alongside COMPACT. (b)(c)(d) show the performance of different distillation methods in Table \ref{['tab:mian-result']} across three student architectures. All results are scaled for better illustration.
• Figure 3: Evolution of dynamic weighting scores for different teachers during training. Teacher 0-3 corresponds to Qwen2.5-72B, Llama3.1-70B, DeepSeek-R1-Llama-70B, and QWQ-32B.
• Figure 4: Layer-wise PCA shift of latent representation in Qwen2.5-1.5B distilled by various methods. COMPACT exhibits minimal shift, confirming its stability in preserving the general capabilities.
• Figure 5: Comprehensive ablation study on model performance.