Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

GenDistiller: Distilling Pre-trained Language Models based on an Autoregressive Generative Model

Yingying Gao, Shilei Zhang, Chao Deng, Junlan Feng

TL;DR

This paper introduces GenDistiller, an autoregressive hidden-layer distillation framework that generates the teacher model's intermediate representations with a compact decoder-only network. By conditioning on the previously generated layer and employing a skip connection plus an output projection, GenDistiller distills layers layer-by-layer without future information, achieving substantial model-size reductions while preserving performance on SUPERB benchmarks. The approach outperforms non-autoregressive distillation baselines and reduces WavLM size by about 82% with only 18% of the original parameters, making high-quality speech representations more deployment-friendly. The findings highlight the importance of autoregressive inter-layer modeling and architectural choices (skip connections, output layer) for effective knowledge distillation in speech models, with practical impact for resource-constrained deployment.

Abstract

Pre-trained speech language models such as HuBERT and WavLM leverage unlabeled speech data for self-supervised learning and offer powerful representations for numerous downstream tasks. Despite the success of these models, their high requirements for memory and computing resource hinder their application on resource restricted devices. Therefore, this paper introduces GenDistiller, a novel knowledge distillation framework which generates the hidden representations of the pre-trained teacher model directly by a much smaller student network. The proposed method takes the previous hidden layer as history and implements a layer-by-layer prediction of the teacher model autoregressively. Experiments on SUPERB reveal the advantage of GenDistiller over the baseline distilling method without an autoregressive framework, with 33% fewer parameters, similar time consumption and better performance on most of the SUPERB tasks. Ultimately, the proposed GenDistiller reduces the size of WavLM by 82%.

GenDistiller: Distilling Pre-trained Language Models based on an Autoregressive Generative Model

TL;DR

This paper introduces GenDistiller, an autoregressive hidden-layer distillation framework that generates the teacher model's intermediate representations with a compact decoder-only network. By conditioning on the previously generated layer and employing a skip connection plus an output projection, GenDistiller distills layers layer-by-layer without future information, achieving substantial model-size reductions while preserving performance on SUPERB benchmarks. The approach outperforms non-autoregressive distillation baselines and reduces WavLM size by about 82% with only 18% of the original parameters, making high-quality speech representations more deployment-friendly. The findings highlight the importance of autoregressive inter-layer modeling and architectural choices (skip connections, output layer) for effective knowledge distillation in speech models, with practical impact for resource-constrained deployment.

Abstract

Pre-trained speech language models such as HuBERT and WavLM leverage unlabeled speech data for self-supervised learning and offer powerful representations for numerous downstream tasks. Despite the success of these models, their high requirements for memory and computing resource hinder their application on resource restricted devices. Therefore, this paper introduces GenDistiller, a novel knowledge distillation framework which generates the hidden representations of the pre-trained teacher model directly by a much smaller student network. The proposed method takes the previous hidden layer as history and implements a layer-by-layer prediction of the teacher model autoregressively. Experiments on SUPERB reveal the advantage of GenDistiller over the baseline distilling method without an autoregressive framework, with 33% fewer parameters, similar time consumption and better performance on most of the SUPERB tasks. Ultimately, the proposed GenDistiller reduces the size of WavLM by 82%.
Paper Structure (15 sections, 2 equations, 1 figure, 5 tables)

This paper contains 15 sections, 2 equations, 1 figure, 5 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Knowledge Distillation for PLMs
  4. Generative Language Models
  5. Method
  6. Model Architecture
  7. Distillation Loss
  8. Experiments
  9. Experimental Setup
  10. Results
  11. SUPERB
  12. Model Size and Inference Speed
  13. Model Architecture
  14. Target Layer
  15. Conclusion

Figures (1)

  • Figure 1: The distillation process and the framework of the proposed GenDistiller.