Teacher-Guided Student Self-Knowledge Distillation Using Diffusion Model

TL;DR

This work tackles the problem of teacher–student distribution mismatch in knowledge distillation by introducing a diffusion-based framework where a teacher classifier guides the denoising of student features. The denoised features serve as a virtual teacher, enabling self-distillation that avoids direct teacher–student feature alignment. The approach combines local feature reconstruction with a global, LSH-guided distillation objective, and it demonstrates strong improvements on image classification and semantic segmentation across diverse backbones and datasets. The findings suggest that teacher-guided diffusion can effectively propagate meaningful, class-related information to the student, enhancing robustness under few-shot and noisy-label conditions and improving generalization in real-world vision tasks.

Abstract

Existing Knowledge Distillation (KD) methods often align feature information between teacher and student by exploring meaningful feature processing and loss functions. However, due to the difference in feature distributions between the teacher and student, the student model may learn incompatible information from the teacher. To address this problem, we propose teacher-guided student Diffusion Self-KD, dubbed as DSKD. Instead of the direct teacher-student alignment, we leverage the teacher classifier to guide the sampling process of denoising student features through a light-weight diffusion model. We then propose a novel locality-sensitive hashing (LSH)-guided feature distillation method between the original and denoised student features. The denoised student features encapsulate teacher knowledge and could be regarded as a teacher role. In this way, our DSKD method could eliminate discrepancies in mapping manners and feature distributions between the teacher and student, while learning meaningful knowledge from the teacher. Experiments on visual recognition tasks demonstrate that DSKD significantly outperforms existing KD methods across various models and datasets. Our code is attached in supplementary material.

Figures (8)

• Figure 1: The overview of our proposed DSKD. We design a highly efficient yet effective diffusion backbone by combining the advantages of U-Net ronneberger2015u and diffusion transformer (DiT) peebles2023scalable. Inspired by DiT, we apply a Multi-Layer Perceptron (MLP) to regress scale and shift parameters for batch normalization layer, and scaling parameters before residual summation, from the conditioned timestep embedding. Inspired by U-Net, we first adopt a $3\times 3$ convolution with stride $S=2$ for downsampling and then a $3\times 3$ deconvolution with stride $S=2$ for upsampling. The final noise prediction is formulated as a residual output.
• Figure 2: Analysis of various guidance strengths on CIFAR-100 and ImageNet datasets.
• Figure 3: Analysis of the weight $\alpha$ of DSKD loss $\mathcal{L}_\mathrm{DSKD}$ on CIFAR-100 and ImageNet datasets.
• Figure 4: Visualizations of the original student features, DiffKD's denoised student features, our denoised student features, and teacher features on ImageNet.
• Figure 5: Illustrations of class probability distributions generated by ResNet-18 pretrained on ImageNet using teacher, our DSKD, DiffKD, and the baseline student, respectively.