Rethinking Knowledge Distillation: A Data Dependent Regulariser With a Negative Asymmetric Payoff

TL;DR

Rethinking Knowledge Distillation reframes KD as a data-dependent regulariser with a negative asymmetric payoff rather than a reliable knowledge-transfer mechanism. By leveraging controlled experiments (including Random Control Distillation and SIDDO) across multimodal tasks and extensive architectures, the study shows that functional transfer is often small and biased toward copying the teacher’s errors, especially as teacher reliance and capacity grow. A gradient-based analysis explains this negative asymmetry, while targeted adversarial transfers demonstrate concrete safety concerns. The results suggest that KD's practical impact is limited and that random-like regularisers can outperform KD on both accuracy and function similarity, calling for auditing of teacher error structures in safety-critical applications. The work advocates reporting functional-transfer metrics alongside traditional accuracy to better assess KD's true effect.

Abstract

Knowledge distillation is often considered a compression mechanism when judged on the resulting student's accuracy and loss, yet its functional impact is poorly understood. In this work, we quantify the compression capacity of knowledge distillation and the resulting knowledge transfer from a functional perspective, decoupling compression from architectural reduction, which provides an improved understanding of knowledge distillation. We employ hypothesis testing, controls, and random control distillation to understand knowledge transfer mechanisms across data modalities. To rigorously test the breadth and limits of our analyses, we explore multiple distillation variants and analyse distillation scaling laws across model sizes. Our findings demonstrate that, while there is statistically significant knowledge transfer in some modalities and architectures, the extent of this transfer is less pronounced than anticipated, even under conditions designed to maximise knowledge sharing. Notably, in cases of significant knowledge transfer, we identify a consistent and severe asymmetric transfer of negative knowledge to the student, raising safety concerns in knowledge distillation applications. Across 12 experimental setups, 9 architectures, and 7 datasets, our findings show that knowledge distillation functions less as a compression mechanism and more as a data-dependent regulariser with a negative asymmetric payoff.

Figures (24)

• Figure 1: Knowledge Distillation Setups.
• Figure 2: Difference in prediction agreement between KD students and the best control baseline on correct (blue) vs. incorrect (red) predictions; error bars show $\pm$1 SEM (SVHN ResNet18).
• Figure 3: Difference in prediction agreement between KD students and the best control baseline on correct (blue) vs. incorrect (red) predictions; error bars show $\pm$1 SEM (TinyImageNet, ResNet-50). Top: base teachers. Bottom: augmented teachers.
• Figure 4: Change in prediction agreement for KD students relative to the best control baseline, decomposed into correct (blue) and incorrect (red) agreement; error bars are $\pm$1 SEM.
• Figure 5: Tiny Shakespeare character distribution.