A Comprehensive Survey on Knowledge Distillation

TL;DR

This survey provides a comprehensive, up-to-date taxonomy of knowledge distillation (KD), organizing methods by knowledge sources (logit-, feature-, and similarity-based), training schemes (offline/online/self), and a broad range of algorithms (attention-based, adversarial, multi-teacher, cross-modal, graph-based, adaptive, and contrastive). It extends KD coverage to new frontiers, including 3D inputs, multi-view data, diffusion models, foundation models, vision-language models, and large language models, with dedicated discussions on open-vocabulary tasks, SAM, and ViT-centered distillation. The authors also discuss practical guidelines for selecting KD strategies, provide performance comparisons across major benchmarks, and highlight ethical, practical, and future directions (notably adaptive/distillation approaches and data-efficient distillation for foundation models and LLMs). Overall, the work consolidates recent advances, proposes novel KD directions (adaptive and contrastive), and underscores KD’s pivotal role in enabling efficient deployment of large, multimodal, and foundation models across diverse domains. The publication serves as a reference for researchers and practitioners aiming to leverage KD to compress, adapt, and extend powerful models while addressing resource and privacy constraints.

Abstract

Deep Neural Networks (DNNs) have achieved notable performance in the fields of computer vision and natural language processing with various applications in both academia and industry. However, with recent advancements in DNNs and transformer models with a tremendous number of parameters, deploying these large models on edge devices causes serious issues such as high runtime and memory consumption. This is especially concerning with the recent large-scale foundation models, Vision-Language Models (VLMs), and Large Language Models (LLMs). Knowledge Distillation (KD) is one of the prominent techniques proposed to address the aforementioned problems using a teacher-student architecture. More specifically, a lightweight student model is trained using additional knowledge from a cumbersome teacher model. In this work, a comprehensive survey of knowledge distillation methods is proposed. This includes reviewing KD from different aspects: distillation sources, distillation schemes, distillation algorithms, distillation by modalities, applications of distillation, and comparison among existing methods. In contrast to most existing surveys, which are either outdated or simply update former surveys, this work proposes a comprehensive survey with a new point of view and representation structure that categorizes and investigates the most recent methods in knowledge distillation. This survey considers various critically important subcategories, including KD for diffusion models, 3D inputs, foundational models, transformers, and LLMs. Furthermore, existing challenges in KD and possible future research directions are discussed. Github page of the project: https://github.com/IPL-Sharif/KD_Survey

Figures (10)

• Figure 1: A general teacher-student framework for knowledge distillation.
• Figure 2: Diagram of the contents of this paper. Knowledge distillation is reviewed from different aspects, including sources, schemes, algorithms, modalities, and applications.
• Figure 3: Illustration of the sources for distillation, including logits, features, and similarities. Logits are the model's last layer output, features are intermediate outputs of the model, and similarities are relationships between features, channels, samples, etc.
• Figure 4: Illustration of distillation schemes. The teacher model in offline distillation is pre-trained and frozen, while in online distillation, the teacher and student are trained simultaneously. In self-distillation, the teacher and student are the same network.
• Figure 5: General overview of a cross-modal distillation method.