Classroom-Inspired Multi-Mentor Distillation with Adaptive Learning Strategies

TL;DR

Problem: fixed mentor–student KD setups struggle with large capacity gaps and error propagation in multi-mentor settings. Approach: ClassroomKD uses a Knowledge Filtering module to rank mentors per input and activate only high-performing ones, and a Mentoring Module that adjusts each active mentor's teaching pace via per-mentor temperature based on the performance gap, with formalizations such as $w^m$, $r^m$, $\Delta r^m$, and $\tau^m$; the distillation loss uses $\mathcal{L}_{distill}(P,Q;\tau)$. Contributions: two modular components, extensive ablations, and demonstrations on CIFAR-100, ImageNet, COCO Keypoints, and MPII showing improvements over state-of-the-art KD methods; analysis includes classroom size, temperature adaptation, and ranking strategies. Significance: dynamic, per-sample mentor selection and adaptive guidance yield stronger knowledge transfer and can reduce computation and energy through more efficient distillation, with potential extension to dataset distillation and broader tasks.

Abstract

We propose ClassroomKD, a novel multi-mentor knowledge distillation framework inspired by classroom environments to enhance knowledge transfer between the student and multiple mentors with different knowledge levels. Unlike traditional methods that rely on fixed mentor-student relationships, our framework dynamically selects and adapts the teaching strategies of diverse mentors based on their effectiveness for each data sample. ClassroomKD comprises two main modules: the Knowledge Filtering (KF) module and the Mentoring module. The KF Module dynamically ranks mentors based on their performance for each input, activating only high-quality mentors to minimize error accumulation and prevent information loss. The Mentoring Module adjusts the distillation strategy by tuning each mentor's influence according to the dynamic performance gap between the student and mentors, effectively modulating the learning pace. Extensive experiments on image classification (CIFAR-100 and ImageNet) and 2D human pose estimation (COCO Keypoints and MPII Human Pose) demonstrate that ClassroomKD outperforms existing knowledge distillation methods for different network architectures. Our results highlight that a dynamic and adaptive approach to mentor selection and guidance leads to more effective knowledge transfer, paving the way for enhanced model performance through distillation.

Figures (21)

• Figure 1: (a) DML: Peer models learn from each other without a hierarchical teacher structure. (b) TAKD: A sequential mentor-student hierarchy with large-to-small knowledge transfer. (c) DGKD: Each mentor teaches all smaller models. (d) ClassroomKD: Our proposed method dynamically selects mentors for each data sample based on the current input and ranks them using the Knowledge Filtering Module. (e) Adaptive Mentoring: The Mentoring Module adjusts teaching strategies of each active mentor according to dynamic rankings, ensuring optimal knowledge transfer.
• Figure 2: The ClassroomKD framework. comprises a Knowledge Filtering (KF) Module and a Mentoring Module. The KF Module optimizes learning by selectively incorporating feedback from higher-ranked mentors, reducing noise transfer and preventing error accumulation. The Mentoring Module adjusts mentor influence based on their performance relative to the student.
• Figure 3: Temperature selection. Grid search using fixed-temperature KD, with the best student performance at $\tau=12$, used as the base temperature in Eq. \ref{['eq:tau_adjustment']}.
• Figure 4: Effect of Classroom Size and Composition. We investigate the effect of mentor count, their architectures, and performance differences on learning.
• Figure 5: Effect of temperature adaption. Our adaptive approach independently adjusts the temperature for each mentor (teacher and peers) over time, allowing them to optimize their teaching strategies dynamically across epochs.