Learning with Less: Knowledge Distillation from Large Language Models via Unlabeled Data

TL;DR

This work tackles the data- and compute-efficiency challenge of knowledge distillation from large language models by distilling into smaller models using unlabeled data. The proposed LLKD framework employs adaptive data selection driven by two signals: teacher confidence for pseudo-label quality and student uncertainty for informativeness, operationalized via dual thresholds $\tau_t^{T}(y)$ and $\tau_t^{S}(y)$ and a weighted loss $\mathcal{L}_{w}$. Across five text-classification datasets, LLKD achieves superior accuracy and data efficiency, including cases where only $3.7\%$ of the training data is used for PubMed-RCT-20k. The method shows robustness to teacher choice (LLaMA or Gemma) and hyper-parameter settings, suggesting practical applicability for reducing labeling costs and computation in real-world deployments of LLM-guided students.

Abstract

In real-world NLP applications, Large Language Models (LLMs) offer promising solutions due to their extensive training on vast datasets. However, the large size and high computation demands of LLMs limit their practicality in many applications, especially when further fine-tuning is required. To address these limitations, smaller models are typically preferred for deployment. However, their training is hindered by the scarcity of labeled data. In contrast, unlabeled data is often readily which can be leveraged by using LLMs to generate pseudo-labels for training smaller models. This enables the smaller models (student) to acquire knowledge from LLMs(teacher) while reducing computational costs. This process introduces challenges, such as potential noisy pseudo-labels. Selecting high-quality and informative data is therefore critical to enhance model performance while improving the efficiency of data utilization. To address this, we propose LLKD that enables Learning with Less computational resources and less data for Knowledge Distillation from LLMs. LLKD is an adaptive sample selection method that incorporates signals from both the teacher and student. Specifically, it prioritizes samples where the teacher demonstrates high confidence in its labeling, indicating reliable labels, and where the student exhibits a high information need, identifying challenging samples that require further learning. Our comprehensive experiments show that LLKD achieves superior performance across various datasets with higher data efficiency.

Figures (7)

• Figure 1: An illustration of the LLKD framework.
• Figure 2: The relationship between teacher model accuracy and teacher confidence (a), and the relationship between student model accuracy and student uncertainty (b) on the validation set of Pubmed-RCT-20k dataset.
• Figure 3: Ablation study on various datasets.
• Figure 4: Teacher ACC and student ACC before and after data selection on the Arxiv-10 dataset.
• Figure 5: Parameter analysis on the Arxiv-10 dataset.