Integrating Knowledge Distillation Methods: A Sequential Multi-Stage Framework

TL;DR

This work tackles the challenge of efficiently distilling knowledge from large teachers into compact students by enabling flexible, stage-wise integration of diverse KD methods. The proposed Sequential Multi-Stage Knowledge Distillation (SMSKD) trains the student across multiple stages, using a frozen reference model from the previous stage to anchor learning and an adaptive true class probability (TCP)-based weighting to balance knowledge retention and integration. Empirical results across CIFAR-100 and Tiny ImageNet with multiple teacher–student pairs show SMSKD consistently outperforms single-source KD and existing multi-source baselines, with ablations confirming the primary benefits stem from stage-wise distillation and reference supervision, while TCP weighting provides supplementary gains. The approach is practical and resource-efficient, introducing negligible overhead and no constraints on which KD methods can be used, making it a versatile solution for combining heterogeneous knowledge sources in deep learning models.

Abstract

Knowledge distillation (KD) transfers knowledge from large teacher models to compact student models, enabling efficient deployment on resource constrained devices. While diverse KD methods, including response based, feature based, and relation based approaches, capture different aspects of teacher knowledge, integrating multiple methods or knowledge sources is promising but often hampered by complex implementation, inflexible combinations, and catastrophic forgetting, which limits practical effectiveness. This work proposes SMSKD (Sequential Multi Stage Knowledge Distillation), a flexible framework that sequentially integrates heterogeneous KD methods. At each stage, the student is trained with a specific distillation method, while a frozen reference model from the previous stage anchors learned knowledge to mitigate forgetting. In addition, we introduce an adaptive weighting mechanism based on the teacher true class probability (TCP) that dynamically adjusts the reference loss per sample to balance knowledge retention and integration. By design, SMSKD supports arbitrary method combinations and stage counts with negligible computational overhead. Extensive experiments show that SMSKD consistently improves student accuracy across diverse teacher student architectures and method combinations, outperforming existing baselines. Ablation studies confirm that stage wise distillation and reference model supervision are primary contributors to performance gains, with TCP based adaptive weighting providing complementary benefits. Overall, SMSKD is a practical and resource efficient solution for integrating heterogeneous KD methods.

Figures (5)

• Figure 1: Overview of the Sequential Multi-Stage Knowledge Distillation Framework
• Figure 2: Results of integrating more diverse distillation methods within the SMSKD framework on CIFAR-100. Each triplet of bars compares the accuracies of two individual methods (left and middle) and their integration under SMSKD (right), demonstrating its ability to incorporate multiple distillation methods.
• Figure 3: Effect of the reference loss weight $\lambda_r$ on AT+KD integration for ResNet56$\rightarrow$ResNet20 on CIFAR-100: (a) Venn diagrams (left side: $\lambda_r=0$, without reference supervision; right side: $\lambda_r \in \{0.1,0.3,0.5,0.8\}$, with reference supervision) show overlap of correctly predicted samples between AT-only and AT+KD models. (b) The bar groups depict model accuracy for each $\lambda_r$ value and the blue curve indicates the IoU between correctly predicted sample sets. As $\lambda_r$ increases, the IoU rises, indicating greater overlap in correctly predicted samples.
• Figure 4: Sensitivity analysis of $\lambda_r$ and the transition epoch on CIFAR-100 using AT+KD integration. Red dashed lines indicate the accuracy of original AT and KD. Teacher-Student: ResNet56-ResNet20.
• Figure 5: Accuracy of the student model across multiple stages with SMSKD. Experiments are conducted on CIFAR-100 using two teacher-student architectures.