Student-Oriented Teacher Knowledge Refinement for Knowledge Distillation

TL;DR

This work reframes knowledge distillation from a teacher-centered paradigm to a student-oriented one, addressing the gap caused by capacity and architectural differences. It introduces SoKD, a plug-in framework consisting of Differentiable Automatic Feature Augmentation (DAFA) and Distinctive Area Detection Module (DAM) to refine the teacher’s knowledge and focus transfer on mutual, informative regions. The method formulates a bi-level optimization where augmentation policies are learned to maximize knowledge transfer effectiveness, while DAM guides distillation to distinctive, task-relevant areas. Experimental results across CIFAR-100, ImageNet, and MS-COCO demonstrate consistent gains over strong baselines, validating SoKD’s generality and practical impact for improving compact models without retraining teachers.

Abstract

Knowledge distillation has become widely recognized for its ability to transfer knowledge from a large teacher network to a compact and more streamlined student network. Traditional knowledge distillation methods primarily follow a teacher-oriented paradigm that imposes the task of learning the teacher's complex knowledge onto the student network. However, significant disparities in model capacity and architectural design hinder the student's comprehension of the complex knowledge imparted by the teacher, resulting in sub-optimal performance. This paper introduces a novel perspective emphasizing student-oriented and refining the teacher's knowledge to better align with the student's needs, thereby improving knowledge transfer effectiveness. Specifically, we present the Student-Oriented Knowledge Distillation (SoKD), which incorporates a learnable feature augmentation strategy during training to refine the teacher's knowledge of the student dynamically. Furthermore, we deploy the Distinctive Area Detection Module (DAM) to identify areas of mutual interest between the teacher and student, concentrating knowledge transfer within these critical areas to avoid transferring irrelevant information. This customized module ensures a more focused and effective knowledge distillation process. Our approach, functioning as a plug-in, could be integrated with various knowledge distillation methods. Extensive experimental results demonstrate the efficacy and generalizability of our method.

Figures (6)

• Figure 1: This research tackles the issue where a powerful teacher network identifies key data regions, but its smaller student struggles to understand these patterns. Our method refines the teacher's knowledge for the student, helping it better recognize these patterns.
• Figure 2: The overall framework of SoKD comprises two key components: 1) DAFA, a differentiable module for augmenting feature strategy search. This module adapts strategies during training, aiming to uncover knowledge more suitable for the student network. 2) DAM, which identifies distinctive areas between the teacher and student networks. This module focuses on areas of mutual interest for knowledge transfer, thereby avoiding unnecessary knowledge distillation.
• Figure 3: DAM in SoKD. Utilizing three head branches, DAM individually predicts the heatmap, size, and offset, thereby identifying the important areas of the feature. The teacher and student features are inputted into the corresponding DAM, which has an identical structure and shared parameters, aiming at identifying distinctive areas of mutual interest to both the teacher and student networks.
• Figure 4: In the distillation process on ImageNet, with ResNet34 serving as the teacher and ResNet18 as the student, the evolution of crucial regions within features. The final results are compared against FitNet and DKD.
• Figure 5: Difference of student and teacher outputs. SoKD leads to a significantly smaller difference than the baseline.