Bayesian Self-Distillation for Image Classification

TL;DR

BSD reframes self-distillation as Recursive Bayesian Estimation over per-sample target distributions, using a Dirichlet prior and discounted Bayesian updates to progressively refine class-probability targets from the model's own predictions. By decoupling targets from hard labels after initialization, BSD improves generalization, calibration, and robustness across CNNs and ViTs on multiple datasets, and exhibits strong resilience to label noise and distribution shifts. The method unifies and extends prior self-distillation approaches, with BSD+ and contrastive loss further boosting robustness under noisy labels. Overall, BSD provides a principled, efficient, single-network alternative that enhances predictive reliability and performance in real-world settings.

Abstract

Supervised training of deep neural networks for classification typically relies on hard targets, which promote overconfidence and can limit calibration, generalization, and robustness. Self-distillation methods aim to mitigate this by leveraging inter-class and sample-specific information present in the model's own predictions, but often remain dependent on hard targets, reducing their effectiveness. With this in mind, we propose Bayesian Self-Distillation (BSD), a principled method for constructing sample-specific target distributions via Bayesian inference using the model's own predictions. Unlike existing approaches, BSD does not rely on hard targets after initialization. BSD consistently yields higher test accuracy (e.g. +1.4% for ResNet-50 on CIFAR-100) and significantly lower Expected Calibration Error (ECE) (-40% ResNet-50, CIFAR-100) than existing architecture-preserving self-distillation methods for a range of deep architectures and datasets. Additional benefits include improved robustness against data corruptions, perturbations, and label noise. When combined with a contrastive loss, BSD achieves state-of-the-art robustness under label noise for single-stage, single-network methods.

Figures (14)

• Figure 1: Bayesian update of a target. At each epoch, the target distributions are updated using the model's own predictions. Here, $\oplus$ denotes the Bayesian update, and $\times$ indicates discounting of the previous posterior parameters by $\gamma \in [0,1]$. See Section \ref{['sec:method']} for details.
• Figure 2: Inter-class component $\bm\mu$ of dark knowledge. Semantical patterns emerge between classes, accentuated by BSD (ResNet-18, CIFAR-10).
• Figure 3: Sample-specific component of dark knowledge. BSD promotes learning of sample-specific information (ResNet-18, CIFAR-10).
• Figure 4: Reliability Diagrams for CIFAR-100 Models. A curve closer to the diagonal indicates better calibration. The models are trained using standard training (baseline), related methods laine2016temporalshen2022selfkim2021self, and the proposed method (BSD).
• Figure 4: Test set accuracy under symmetric and asymmetric label noise. Performance of lightweight methods (top, ResNet-18) szegedy2016rethinkingwang2019symmetriczhang2017mixuplaine2016temporal and state-of-the-art methods (bottom, PreAct ResNet-18) feng2023otchang2023csotxu2025revisiting on CIFAR-10. The best results are highlighted in bold.