Improve Knowledge Distillation via Label Revision and Data Selection

TL;DR

This work tackles the problem that teacher predictions in knowledge distillation can be unreliable and mislead the student. It introduces Label Revision (LR), which forms revised soft labels via p = β p^t + (1−β) y with a constraint β < 1/(p^t_max − p^t_tar + 1) and β = η/(p^t_max − p^t_tar + 1), and Data Selection (DS), which uses influence-based scoring to only distill a subset D^t of the data while supervising the rest with ground-truth. The method can be plugged into existing KD approaches and consistently improves performance on CIFAR-100 and ImageNet across diverse teacher–student pairs, while also reducing training cost relative to feature-based distillation. Key findings show that LR and DS together yield the most significant gains, are robust to hyperparameters like λ1, λ2, and η, and extend to large-scale settings with reliable improvements.

Abstract

Knowledge distillation (KD) has become a widely used technique in the field of model compression, which aims to transfer knowledge from a large teacher model to a lightweight student model for efficient network development. In addition to the supervision of ground truth, the vanilla KD method regards the predictions of the teacher as soft labels to supervise the training of the student model. Based on vanilla KD, various approaches have been developed to further improve the performance of the student model. However, few of these previous methods have considered the reliability of the supervision from teacher models. Supervision from erroneous predictions may mislead the training of the student model. This paper therefore proposes to tackle this problem from two aspects: Label Revision to rectify the incorrect supervision and Data Selection to select appropriate samples for distillation to reduce the impact of erroneous supervision. In the former, we propose to rectify the teacher's inaccurate predictions using the ground truth. In the latter, we introduce a data selection technique to choose suitable training samples to be supervised by the teacher, thereby reducing the impact of incorrect predictions to some extent. Experiment results demonstrate the effectiveness of our proposed method, and show that our method can be combined with other distillation approaches, improving their performance.

Figures (3)

• Figure 1: An overview of training procedure with our proposed Label Revision (LR) and Data Selection (DS). For the entire training set, it is first split into two parts through DS, i.e., Set I and Set II. For Set I, it is input to the student model, then Loss 1 is calculated as the cross-entropy loss between the obtained Logits $\text{S}_\text{I}$ and Label $\text{y}_\text{I}$. With respect to Set II, it is input to the student and teacher at the same time, where the teacher's predictions Logits T is revised via LR before computing the distillation loss Loss 3. Similar to Loss 1, Loss 2 is the cross-entropy loss between Logits $\text{S}_\text{II}$ and Label $\text{y}_\text{II}$. Finally, the whole loss is a weighted summation of the three losses. More details about the calculation of losses are provided in Section \ref{['sec:method']}.
• Figure 2: A four-class example of Label Revision.
• Figure 3: Comparison of training time and accuracy on CIFAR-100. The teacher and student are ResNet32x4 and ResNet8x4, respectively.