Knowledge Distillation and Dataset Distillation of Large Language Models: Emerging Trends, Challenges, and Future Directions

TL;DR

This survey analyzes Knowledge Distillation (KD) and Dataset Distillation (DD) as complementary strategies to compress Large Language Models (LLMs) while retaining their reasoning, linguistic diversity, and domain competencies. It surveys KD methodologies (rationale-based, uncertainty-aware, multi-teacher, dynamic/adaptive, vision-centric, task-specific, and theory) and DD techniques (optimization-based and synthetic-data generation), then discusses integrating KD and DD and their evaluation. The work highlights applications across healthcare, education, and bioinformatics, underlining benefits for efficiency and deployment, while acknowledging challenges in preserving emergent reasoning, data diversity, and evaluation standards. It also outlines future directions, including trustworthiness, fairness, dynamic teacher/data evolution, and architecture-aware distillation, to enable scalable, responsible, and data-efficient LLMs.

Abstract

The exponential growth of Large Language Models (LLMs) continues to highlight the need for efficient strategies to meet ever-expanding computational and data demands. This survey provides a comprehensive analysis of two complementary paradigms: Knowledge Distillation (KD) and Dataset Distillation (DD), both aimed at compressing LLMs while preserving their advanced reasoning capabilities and linguistic diversity. We first examine key methodologies in KD, such as task-specific alignment, rationale-based training, and multi-teacher frameworks, alongside DD techniques that synthesize compact, high-impact datasets through optimization-based gradient matching, latent space regularization, and generative synthesis. Building on these foundations, we explore how integrating KD and DD can produce more effective and scalable compression strategies. Together, these approaches address persistent challenges in model scalability, architectural heterogeneity, and the preservation of emergent LLM abilities. We further highlight applications across domains such as healthcare and education, where distillation enables efficient deployment without sacrificing performance. Despite substantial progress, open challenges remain in preserving emergent reasoning and linguistic diversity, enabling efficient adaptation to continually evolving teacher models and datasets, and establishing comprehensive evaluation protocols. By synthesizing methodological innovations, theoretical foundations, and practical insights, our survey charts a path toward sustainable, resource-efficient LLMs through the tighter integration of KD and DD principles.

Figures (10)

• Figure 1: Taxonomy of Distillation of Large Language Models.
• Figure 2: Overview of Knowledge Distillation in LLMs. Knowledge is distilled from a teacher LLM, which has been trained on a large existing database. This knowledge, potentially enriched with current, task-specific data, is transferred to a smaller student LLM. By learning from both the teacher’s guidance and the current data, the student LLM becomes more efficient and effective at performing downstream tasks.
• Figure 3: Overview of Dataset Distillation in LLMs. A teacher LLM is trained on a massive original database. Through dataset distillation, a compact, high-quality subset (Distilled Database) is synthesized to preserve essential knowledge. This smaller dataset is then used to train a student LLM, aiming to achieve similar performance as the teacher while requiring significantly fewer data.
• Figure 4: Bayesian Knowledge Distillation. Knowledge distilled from the teacher LLM is incorporated as a teacher-informed prior over the student LLM’s parameters. This leads to a posterior distribution that equips the student LLM with uncertainty quantification capabilities.
• Figure 5: Multi-Teacher Knowledge Distillation. The variants of Multi-Teacher KD include different weighting strategies and teacher settings.