Revisiting Knowledge Distillation: The Hidden Role of Dataset Size

TL;DR

This work reframes knowledge distillation (KD) by varying dataset size rather than architecture alone, revealing a data efficiency that is amplified in low-data regimes. Using a controlled setup where a teacher trained on $N$ samples guides a student trained on $M$ samples with relative fraction $\kappa = M/N$, the authors show substantial performance gains (up to around 10–25% in accuracy or perplexity improvements) for $\kappa < 1$ across vision and language tasks and various architectures. They systematically test existing KD theories—dark knowledge, label smoothing, and fidelity—and find label smoothing insufficient to explain the observed data-efficient behavior, while evidence for dark knowledge and fidelity remains, with nuances across tasks. The study also analyzes how factors such as temperature, target type (soft vs hard), and the relative size of teacher and student modulate the gains, and demonstrates that a clear data-amount signal governs when KD is advantageous. Overall, the dataset size emerges as a fundamental variable in KD, suggesting directions for theoretical grounding and practical use in data-scarce settings, albeit with computational-cost considerations due to teacher inferences.

Abstract

The concept of knowledge distillation (KD) describes the training of a student model from a teacher model and is a widely adopted technique in deep learning. However, it is still not clear how and why distillation works. Previous studies focus on two central aspects of distillation: model size, and generalisation. In this work we study distillation in a third dimension: dataset size. We present a suite of experiments across a wide range of datasets, tasks and neural architectures, demonstrating that the effect of distillation is not only preserved but amplified in low-data regimes. We call this newly discovered property the data efficiency of distillation. Equipped with this new perspective, we test the predictive power of existing theories of KD as we vary the dataset size. Our results disprove the hypothesis that distillation can be understood as label smoothing, and provide further evidence in support of the dark knowledge hypothesis. Finally, we analyse the impact of modelling factors such as the objective, scale and relative number of samples on the observed phenomenon. Ultimately, this work reveals that the dataset size may be a fundamental but overlooked variable in the mechanisms underpinning distillation.

Figures (25)

• Figure 1: Overview of our study. (Left and Center) Schematic summary of the contributions of this work. Prior research has primarily emphasized two main benefits of knowledge distillation—model compression and improved generalization. We introduce a third, previously underexplored dimension: data efficiency, showing that distillation yields the largest relative gains in low-data regimes. (Right) Experimental setup used throughout the paper. Students share the same architecture as the teacher or are smaller, and are trained with varying dataset fractions and temperatures. We systematically compare training with one-hot labels versus distillation targets to isolate the role of soft supervision across data scales.
• Figure 2: Knowledge distillation improves sample efficiency. Test error (for image classification) and perplexity (for autoregressive language modelling) as a function of the relative training dataset size $\kappa$, averaged over $5$ seeds. We compare models of the same architecture trained with either label training or knowledge distillation. Distillation consistently dominates label training in the low data regime.
• Figure 3: Test perplexity over FLOPs for GPT-MINI students trained with either distillation or label training, varying teacher size.
• Figure 4: Distillation induces feature kernel alignment in image classification settings. On the y-axes the CKA of the feature kernels $k_\phi$ of the KD and LT students to the teacher's feature kernel. Note that the LT students and the LT teacher are both trained with labels. On the x-axis the portion of dataset used. We observe that KD produces markedly steeper curves, yielding high feature kernel alignments at low $\kappa$.
• Figure 5: Distilled students form a compact cluster. Feature-kernel alignment between students trained with distillation or one-hot labels on the same fraction of CIFAR10 from three different initializations (six networks total). Entries correspond to pairwise alignments; the first three rows and columns represent distilled students, the last three label-trained students. From left to right: $\kappa=0.02$, $\kappa=0.1$, and $\kappa=0.2$. Temperature $\tau=20$ in all plots.