MIND: From Passive Mimicry to Active Reasoning through Capability-Aware Multi-Perspective CoT Distillation

TL;DR

The paper addresses the challenge of transferring Chain-of-Thought reasoning from large LLMs to smaller models while preserving cross-domain generalization. It introduces MIND, a capability-aware distillation framework that synthesizes eight diverse reasoning perspectives via a Teaching Assistant, MetaNet, and a Feedback-Driven Inertia Calibration mechanism to adapt supervision to the student’s evolving capacity. Through multi-perspective data construction, latent-space analysis, and adaptive fusion with consistency regularization, MIND achieves state-of-the-art performance on both in-distribution and out-of-distribution benchmarks and demonstrates robust internalization of reasoning primitives. The results highlight practical benefits for deploying compact models with universal reasoning capabilities, and the latent-space findings justify the approach by showing topologically separable cognitive primitives rather than surface-level imitation.

Abstract

While Large Language Models (LLMs) have emerged with remarkable capabilities in complex tasks through Chain-of-Thought reasoning, practical resource constraints have sparked interest in transferring these abilities to smaller models. However, achieving both domain performance and cross-domain generalization remains challenging. Existing approaches typically restrict students to following a single golden rationale and treat different reasoning paths independently. Due to distinct inductive biases and intrinsic preferences, alongside the student's evolving capacity and reasoning preferences during training, a teacher's "optimal" rationale could act as out-of-distribution noise. This misalignment leads to a degeneration of the student's latent reasoning distribution, causing suboptimal performance. To bridge this gap, we propose MIND, a capability-adaptive framework that transitions distillation from passive mimicry to active cognitive construction. We synthesize diverse teacher perspectives through a novel "Teaching Assistant" network. By employing a Feedback-Driven Inertia Calibration mechanism, this network utilizes inertia-filtered training loss to align supervision with the student's current adaptability, effectively enhancing performance while mitigating catastrophic forgetting. Extensive experiments demonstrate that MIND achieves state-of-the-art performance on both in-distribution and out-of-distribution benchmarks, and our sophisticated latent space analysis further confirms the mechanism of reasoning ability internalization.

Figures (8)

• Figure 1: Reasoning Perspectives inherent in LLMs. We only demonstrate the distinctive ones used to clearly demonstrate our distillation method. Selectively using a subset for efficiency does not sacrifice performance.
• Figure 2: Demonstration of the dataset construction pipeline. We adopt a Judge LLM to identify reasoning with poor quality or leading to wrong answers, and maintain the consistency between rationales and the correct answer.
• Figure 3: Overview of our method MIND. (1) We first prompt the teacher model to generate multi-perspective rationales. (2) Then, we warm up the MetaNet for a few steps on the acquired dataset and recalibrate it using the student's performance feedback. (3) We train the student model with SFT supervision and consistency regularization.
• Figure 4: Methodology and results of the Latent Space Visualization Analysis. (a) and (b)demonstrate the mechanism, (c) illustrates the semantic level separation of LLM's reasoning paths, (d) provides a rigorous visualization of the differentiation of specifically trained students' latent space.
• Figure 5: Topological comparison of the student's latent reasoning manifold. (a) The Vanilla CoT distilled student exhibits severe mode collapse. (b) The MIND-distilled student demonstrates comprehensive coverage.