DDK: Distilling Domain Knowledge for Efficient Large Language Models

TL;DR

DDK addresses the inefficiency of standard KD in LLM distillation by explicitly modeling domain-wise performance gaps between teacher and student. It introduces a domain discrepancy factor $\mathbf{r} \in \mathbb{R}^N$ and a domain knowledge guided sampling strategy, augmented with a factor-smooth updating mechanism to stabilize data composition across domains. The training objective combines cross-entropy on distillation data with a KL-divergence term between softened teacher and student logits at temperature $T$, balanced by $\gamma$. Empirical results on multiple teacher-student pairs (e.g., Qwen-1.5 and LLaMA2) across diverse domains show that DDK consistently outperforms strong baselines, improves reasoning and code-related tasks, and generalizes to Code LLMs and different configurations, indicating practical benefits for deploying compact, high-performing LLMs.

Abstract

Despite the advanced intelligence abilities of large language models (LLMs) in various applications, they still face significant computational and storage demands. Knowledge Distillation (KD) has emerged as an effective strategy to improve the performance of a smaller LLM (i.e., the student model) by transferring knowledge from a high-performing LLM (i.e., the teacher model). Prevailing techniques in LLM distillation typically use a black-box model API to generate high-quality pretrained and aligned datasets, or utilize white-box distillation by altering the loss function to better transfer knowledge from the teacher LLM. However, these methods ignore the knowledge differences between the student and teacher LLMs across domains. This results in excessive focus on domains with minimal performance gaps and insufficient attention to domains with large gaps, reducing overall performance. In this paper, we introduce a new LLM distillation framework called DDK, which dynamically adjusts the composition of the distillation dataset in a smooth manner according to the domain performance differences between the teacher and student models, making the distillation process more stable and effective. Extensive evaluations show that DDK significantly improves the performance of student models, outperforming both continuously pretrained baselines and existing knowledge distillation methods by a large margin.

Figures (3)

• Figure 1: The perplexity scores of different methods across different domains for different methods (See Section \ref{['sec:expr']} for more details.). Note that "Chinese CC" denotes "Chinese CommonCrawl".
• Figure 2: Overview of the distillation process of DDK. First, the training dataset is divided into distinct domains based on predefined criteria. Then, DDK dynamically modulates the distribution of domain-specific data, augmenting the amount allocated to domains where the student model struggles the most. The proportions attributed to each domain are recalculated at distillation intervals by employing a factor smooth updating approach.
• Figure 3: Visualization on the domain discrepancy among three domains.