Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Survey on Knowledge Distillation for Large Language Models: Methods, Evaluation, and Application

Chuanpeng Yang, Wang Lu, Yao Zhu, Yidong Wang, Qian Chen, Chenlong Gao, Bingjie Yan, Yiqiang Chen

TL;DR

This survey analyzes knowledge distillation (KD) for large language models (LLMs) through three lenses: methods, evaluation, and application, distinguishing white-box KD (logits-based and hint-based) from black-box KD (in-context learning, chain-of-thought, and instruction following). It surveys robustness and practicality, highlighting how distillation scales with model size, data availability, and API access, and it surveys real-world applications in healthcare, education, and law. Key contributions include synthesizing state-of-the-art KD techniques for LLMs, comparing white-box and black-box paradigms, and outlining unified benchmarks and interpretability challenges to guide future work. The practical impact lies in guiding researchers and practitioners to select and design KD schemes that balance compression, performance, and deployment constraints across diverse domains.

Abstract

Large Language Models (LLMs) have showcased exceptional capabilities in various domains, attracting significant interest from both academia and industry. Despite their impressive performance, the substantial size and computational demands of LLMs pose considerable challenges for practical deployment, particularly in environments with limited resources. The endeavor to compress language models while maintaining their accuracy has become a focal point of research. Among the various methods, knowledge distillation has emerged as an effective technique to enhance inference speed without greatly compromising performance. This paper presents a thorough survey from three aspects: method, evaluation, and application, exploring knowledge distillation techniques tailored specifically for LLMs. Specifically, we divide the methods into white-box KD and black-box KD to better illustrate their differences. Furthermore, we also explored the evaluation tasks and distillation effects between different distillation methods, and proposed directions for future research. Through in-depth understanding of the latest advancements and practical applications, this survey provides valuable resources for researchers, paving the way for sustained progress in this field.

Survey on Knowledge Distillation for Large Language Models: Methods, Evaluation, and Application

TL;DR

This survey analyzes knowledge distillation (KD) for large language models (LLMs) through three lenses: methods, evaluation, and application, distinguishing white-box KD (logits-based and hint-based) from black-box KD (in-context learning, chain-of-thought, and instruction following). It surveys robustness and practicality, highlighting how distillation scales with model size, data availability, and API access, and it surveys real-world applications in healthcare, education, and law. Key contributions include synthesizing state-of-the-art KD techniques for LLMs, comparing white-box and black-box paradigms, and outlining unified benchmarks and interpretability challenges to guide future work. The practical impact lies in guiding researchers and practitioners to select and design KD schemes that balance compression, performance, and deployment constraints across diverse domains.

Abstract

Large Language Models (LLMs) have showcased exceptional capabilities in various domains, attracting significant interest from both academia and industry. Despite their impressive performance, the substantial size and computational demands of LLMs pose considerable challenges for practical deployment, particularly in environments with limited resources. The endeavor to compress language models while maintaining their accuracy has become a focal point of research. Among the various methods, knowledge distillation has emerged as an effective technique to enhance inference speed without greatly compromising performance. This paper presents a thorough survey from three aspects: method, evaluation, and application, exploring knowledge distillation techniques tailored specifically for LLMs. Specifically, we divide the methods into white-box KD and black-box KD to better illustrate their differences. Furthermore, we also explored the evaluation tasks and distillation effects between different distillation methods, and proposed directions for future research. Through in-depth understanding of the latest advancements and practical applications, this survey provides valuable resources for researchers, paving the way for sustained progress in this field.
Paper Structure (30 sections, 5 equations, 1 figure, 9 tables)

This paper contains 30 sections, 5 equations, 1 figure, 9 tables.

Table of Contents

  1. Introduction
  2. Overview of Knowledge Distillation
  3. Logits-based KD
  4. Hint-based KD
  5. In-Context Learning
  6. Chain-of-Thought
  7. Instruction Following
  8. Knowledge Distillation in Large Language Models
  9. White-box Knowledge Distillation
  10. Logits-based KD
  11. Hint-based KD
  12. Robustness Evaluation of White-box KD
  13. Discussion on White-box KD
  14. Black-box Knowledge Distillation
  15. In-Context Learning
  16. ...and 15 more sections

Figures (1)

  • Figure 1: The general taxonomy framework of this survey.