Self-Distillation Amplifies Regularization in Hilbert Space

TL;DR

The paper analyzes self-distillation within a nonlinear regression framework in a Hilbert space with $\ell_2$ regularization, showing that iterative distillation acts as a nonlinear power iteration that sparsifies the basis via the diagonal product $\boldsymbol{B}_t$. It provides lower bounds on the number of meaningful rounds before collapse and demonstrates that self-distillation increases sparsity as rounds proceed, with near-interpolation ($\epsilon\to0$) regimes yielding stronger sparsity. A kernel-ridge/generalization bridge is established through an analytically related kernel $g^{\dag}$, enabling existing generalization bounds to apply to self-distilled predictors. The results are supported by illustrative examples and NTK-based experiments on neural networks, showing initial generalization gains that can reverse with excessive distillation due to over-regularization, linking theory to observed deep-learning phenomena.

Abstract

Knowledge distillation introduced in the deep learning context is a method to transfer knowledge from one architecture to another. In particular, when the architectures are identical, this is called self-distillation. The idea is to feed in predictions of the trained model as new target values for retraining (and iterate this loop possibly a few times). It has been empirically observed that the self-distilled model often achieves higher accuracy on held out data. Why this happens, however, has been a mystery: the self-distillation dynamics does not receive any new information about the task and solely evolves by looping over training. To the best of our knowledge, there is no rigorous understanding of this phenomenon. This work provides the first theoretical analysis of self-distillation. We focus on fitting a nonlinear function to training data, where the model space is Hilbert space and fitting is subject to $\ell_2$ regularization in this function space. We show that self-distillation iterations modify regularization by progressively limiting the number of basis functions that can be used to represent the solution. This implies (as we also verify empirically) that while a few rounds of self-distillation may reduce over-fitting, further rounds may lead to under-fitting and thus worse performance.

Figures (10)

• Figure 1: Schematic illustration of the self-distillation process for two iterations.
• Figure 2: Example with $R(f)(x) \triangleq \int_0^1 ( \frac{d^2}{dx^2} f(x) )^2 \, dx$. (a) Green's function associated with the kernel of $R$. (b) Noisy training samples (blue dots) from underlying function (orange) $y=\sin(2\pi x)$. Fitting without regularization leads to overfitting (blue curve). (c) Four rounds of self-distillation (blue, orange, green, red) with $\epsilon=0.04$.
• Figure 3: Evolution of the diagonal entries of (the diagonal matrix) $\boldsymbol{B}_t$ from (\ref{['eq:def_B']}) at distillation rounds $t=0$ (left most) to $t=3$ (right most). The number of training points is $K=11$, so $\boldsymbol{B}_t$ which is $K \times K$ diagonal matrix has $11$ entries on its diagonal, each corresponding to one of the bars in the chart.
• Figure 4: Accuracy of self-distillation steps using Resnet with $\ell_2$ loss on neural network predictions. (Left 2 plots): test/train accuracy on CIFAR-10. (Right 2 plots): test/train accuracy on CIFAR-100.
• Figure 5: Self-distillation steps using Resnet with cross-entropy loss on neural network predictions. (Left 2 plots): test/train accuracy on CIFAR-10. (Right 2 plots): test/train accuracy on CIFAR-100.

Theorems & Definitions (12)

• Proposition 1
• Proposition 2
• Proposition 3
• Proposition 4
• Theorem 5
• Theorem 6
• Proposition 1
• Proposition 2
• Proposition 3
• Proposition 4