Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Meta-KD: A Meta Knowledge Distillation Framework for Language Model Compression across Domains

Haojie Pan, Chengyu Wang, Minghui Qiu, Yichang Zhang, Yaliang Li, Jun Huang

TL;DR

Meta-KD presents a cross-domain knowledge distillation framework that first trains a meta-teacher on multiple domains to capture transferable instance- and feature-level knowledge, then distills this knowledge into domain-specific students. It introduces prototype-based instance weighting and a domain-adversarial objective for the meta-teacher, plus a domain-expertise weighted and transferable-knowledge KD loss to guide students. Experiments on MNLI and Amazon Reviews show Meta-KD achieves state-of-the-art or near state-of-the-art performance with substantially smaller models, especially when in-domain data are scarce or unavailable for meta-teacher learning. This work offers a principled approach to PLM compression that leverages cross-domain information to improve generalization and practicality in real-world deployment.

Abstract

Pre-trained language models have been applied to various NLP tasks with considerable performance gains. However, the large model sizes, together with the long inference time, limit the deployment of such models in real-time applications. One line of model compression approaches considers knowledge distillation to distill large teacher models into small student models. Most of these studies focus on single-domain only, which ignores the transferable knowledge from other domains. We notice that training a teacher with transferable knowledge digested across domains can achieve better generalization capability to help knowledge distillation. Hence we propose a Meta-Knowledge Distillation (Meta-KD) framework to build a meta-teacher model that captures transferable knowledge across domains and passes such knowledge to students. Specifically, we explicitly force the meta-teacher to capture transferable knowledge at both instance-level and feature-level from multiple domains, and then propose a meta-distillation algorithm to learn single-domain student models with guidance from the meta-teacher. Experiments on public multi-domain NLP tasks show the effectiveness and superiority of the proposed Meta-KD framework. Further, we also demonstrate the capability of Meta-KD in the settings where the training data is scarce.

Meta-KD: A Meta Knowledge Distillation Framework for Language Model Compression across Domains

TL;DR

Meta-KD presents a cross-domain knowledge distillation framework that first trains a meta-teacher on multiple domains to capture transferable instance- and feature-level knowledge, then distills this knowledge into domain-specific students. It introduces prototype-based instance weighting and a domain-adversarial objective for the meta-teacher, plus a domain-expertise weighted and transferable-knowledge KD loss to guide students. Experiments on MNLI and Amazon Reviews show Meta-KD achieves state-of-the-art or near state-of-the-art performance with substantially smaller models, especially when in-domain data are scarce or unavailable for meta-teacher learning. This work offers a principled approach to PLM compression that leverages cross-domain information to improve generalization and practicality in real-world deployment.

Abstract

Pre-trained language models have been applied to various NLP tasks with considerable performance gains. However, the large model sizes, together with the long inference time, limit the deployment of such models in real-time applications. One line of model compression approaches considers knowledge distillation to distill large teacher models into small student models. Most of these studies focus on single-domain only, which ignores the transferable knowledge from other domains. We notice that training a teacher with transferable knowledge digested across domains can achieve better generalization capability to help knowledge distillation. Hence we propose a Meta-Knowledge Distillation (Meta-KD) framework to build a meta-teacher model that captures transferable knowledge across domains and passes such knowledge to students. Specifically, we explicitly force the meta-teacher to capture transferable knowledge at both instance-level and feature-level from multiple domains, and then propose a meta-distillation algorithm to learn single-domain student models with guidance from the meta-teacher. Experiments on public multi-domain NLP tasks show the effectiveness and superiority of the proposed Meta-KD framework. Further, we also demonstrate the capability of Meta-KD in the settings where the training data is scarce.

Paper Structure

This paper contains 19 sections, 8 equations, 4 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Knowledge Distillation (KD)
  4. PLM Compression
  5. Transfer Learning and Meta-learning
  6. The Meta-KD Framework
  7. An Overview of Meta-KD
  8. Meta-teacher Learning
  9. Meta-distillation
  10. Experiments
  11. Tasks and Datasets
  12. Baselines
  13. Implementation Details
  14. Experimental Results
  15. Ablation Study
  16. ...and 4 more sections

Figures (4)

  • Figure 1: A motivation example of academic learning. A physics student may learn physics equations better with a powerful all-purpose teacher.
  • Figure 2: An overview of meta-distillation and the neural architecture that we adopt for knowledge distillation.
  • Figure 3: Improvement rate w.r.t different portion (sample rate) of training data in usage.
  • Figure 4: Model performance w.r.t. the transferable KD factor $\gamma_2$