Why Knowledge Distillation Works in Generative Models: A Minimal Working Explanation

TL;DR

Knowledge distillation in generative modeling improves sample quality by biasing the student toward high-density regions of the ground-truth distribution as the teacher becomes more selective. The authors formulate a minimal working explanation using a Gaussian-mixture data-generating process with a temperature-like parameter that controls teacher entropy, revealing a precision–recall trade-off between $Precision(\beta)$ and $Recall(\beta)$. This dynamic is demonstrated in a toy simulation and replicated in large autoregressive language models (SmolLM2), where lower-entropy teachers yield sharper, higher-precision generations at the cost of reduced coverage. The results offer a principled design intuition for when KD should be favored (e.g., instruction tuning) and how to tune training signals via teacher selectivity.

Abstract

Knowledge distillation (KD) is a core component in the training and deployment of modern generative models, particularly large language models (LLMs). While its empirical benefits are well documented -- enabling smaller student models to emulate the performance of much larger teachers -- the underlying mechanisms by which KD improves generative quality remain poorly understood. In this work, we present a minimal working explanation of KD in generative modeling. Using a controlled simulation with mixtures of Gaussians, we demonstrate that distillation induces a trade-off between precision and recall in the student model. As the teacher distribution becomes more selective, the student concentrates more probability mass on high-likelihood regions at the expense of coverage -- a behavior modulated by a single entropy-controlling parameter. We then validate this effect in a large-scale language modeling setup using the SmolLM2 family of models. Empirical results reveal the same precision-recall dynamics observed in simulation, where precision corresponds to sample quality and recall to distributional coverage. This precision-recall trade-off in LLMs is found to be especially beneficial in scenarios where sample quality is more important than diversity, such as instruction tuning or downstream generation. Our analysis provides a simple and general explanation for the effectiveness of KD in generative modeling.

Figures (5)

• Figure 1: (a) Contours show the probability density of the ground-truth distribution $p^*$, with dots representing samples drawn from it. (b) The contours correspond to the student model trained directly without distillation.
• Figure 2: (a) Contour plot of the teacher distribution $p'$, with samples from the true distribution $p^*$ overlaid as blue dots. (b) Samples (green) drawn from the $\beta$-modulated teacher distribution $p'(x; \theta', \beta=100)$, showing strong concentration on the bottom three modes of $p^*$. (c) Contours of the student models: the dashed black contour corresponds to a student trained directly on $p^*$ samples, while the green contour represents a student trained on teacher samples (distillation). The distilled student clearly focuses on a narrower region emphasized by the teacher. This setup corresponds to a low-difficulty case: with $\beta = 100$, the teacher concentrates on a single dominant component ($\alpha'_k \approx 1$), and the student has just enough capacity ($K" = 1$) to match it, resulting in the Difficulty measure close to 0.
• Figure 3: Overview of our LLM distillation setup. We first treat the pretrained SmolLM2 1.7B model as the ground-truth distribution $p^*$ and sample 10M sequences to construct dataset $D$. We then pretrain a 360M teacher model $p'$ on $D$ using next-token prediction loss. To control the teacher's entropy, we sample from $p'$ with varying temperature values $\tau$ to generate distillation datasets $D'_\tau$. Finally, we train a 135M student model $p"$ on each $D'_\tau$ and evaluate its precision and recall with respect to $p^*$.
• Figure 4: Score distribution of Precision (left) and Recall (right) based on the $\tau$ parameter and the $P"(D)$ model. Each box plot illustrates the interquartile range across five seeds, with the orange line indicating the arithmetic mean. Higher (less negative) values on the y-axis denote better result.
• Figure 5: 2D UMAP projections of sentence embeddings generated by the ground-truth model $p^*$ (gray) and student models $p"(D'_\tau)$ (red) trained via distillation with varying teacher sampling temperatures $\tau$. As $\tau$ increases, the student's output distribution becomes more dispersed in the embedding space, covering a broader portion of the support of $p^*$. Conversely, lower $\tau$ values result in tighter clustering around specific regions, reflecting the student's emphasis on high-likelihood modes guided by the teacher's selectivity.