Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Distillation-based Layer Dropping (DLD): Effective End-to-end Framework for Dynamic Speech Networks

Abdul Hannan, Daniele Falavigna, Shah Nawaz, Mubashir Noman, Markus Schedl, Alessio Brutti

TL;DR

Dynamic speech recognition on edge devices requires adaptable models, but traditional layer dropping often harms performance across dropping levels. The authors propose Distillation-based Layer Dropping (DLD), an end-to-end framework that distills knowledge from a full teacher into a dynamic student by aligning latent embeddings with a loss $L_{KLD}$ and training with a transcription objective $L_{CTC}$, with total loss $L = L_{KLD} + L_{CTC}$. Across Conformer and WavLM backbones on LibriSpeech-1000 and TED-LIUM v3, DLD delivers superior dynamic performance, improving WER and enabling no-dropping accuracy while reducing training time. The approach shows faster convergence, stronger performance under varying encoder depth, and generalizes to other downstream tasks such as FSC, making it practical for resource-constrained ASR deployments.

Abstract

Edge devices operate in constrained and varying resource settings, requiring dynamic architectures that can adapt to limitations of the available resources. To meet such demands, layer dropping ($\mathcal{LD}$) approach is typically used to transform static models into dynamic ones by skipping parts of the network along with reducing overall computational complexity. However, existing $\mathcal{LD}$ methods greatly impact the dynamic model's performance for low and high dropping cases, deteriorating the performance-computation trade-off. To this end, we propose a distillation-based layer dropping (DLD) framework that effectively combines the capabilities of knowledge distillation and $\mathcal{LD}$ in an end-to-end fashion, thereby achieving state-of-the-art performance for dynamic speech networks. Comprehensive experimentation utilizing well-known speech recognition methods, including conformer and WavLM, on three public benchmarks demonstrates the effectiveness of our framework, reducing the word error rate by $9.32\%$ and $2.25\%$ for high and no dropping cases with $33.3\%$ reduction in training time.

Distillation-based Layer Dropping (DLD): Effective End-to-end Framework for Dynamic Speech Networks

TL;DR

Dynamic speech recognition on edge devices requires adaptable models, but traditional layer dropping often harms performance across dropping levels. The authors propose Distillation-based Layer Dropping (DLD), an end-to-end framework that distills knowledge from a full teacher into a dynamic student by aligning latent embeddings with a loss and training with a transcription objective , with total loss . Across Conformer and WavLM backbones on LibriSpeech-1000 and TED-LIUM v3, DLD delivers superior dynamic performance, improving WER and enabling no-dropping accuracy while reducing training time. The approach shows faster convergence, stronger performance under varying encoder depth, and generalizes to other downstream tasks such as FSC, making it practical for resource-constrained ASR deployments.

Abstract

Edge devices operate in constrained and varying resource settings, requiring dynamic architectures that can adapt to limitations of the available resources. To meet such demands, layer dropping () approach is typically used to transform static models into dynamic ones by skipping parts of the network along with reducing overall computational complexity. However, existing methods greatly impact the dynamic model's performance for low and high dropping cases, deteriorating the performance-computation trade-off. To this end, we propose a distillation-based layer dropping (DLD) framework that effectively combines the capabilities of knowledge distillation and in an end-to-end fashion, thereby achieving state-of-the-art performance for dynamic speech networks. Comprehensive experimentation utilizing well-known speech recognition methods, including conformer and WavLM, on three public benchmarks demonstrates the effectiveness of our framework, reducing the word error rate by and for high and no dropping cases with reduction in training time.
Paper Structure (11 sections, 4 equations, 3 figures, 3 tables)

This paper contains 11 sections, 4 equations, 3 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Methodology
  3. Experimentation and Results
  4. Results
  5. Conformer
  6. WavLM
  7. Ablations
  8. Framework performance evaluation as a function of training time
  9. Generalization of proposed framework on other downstream tasks
  10. Conclusion
  11. Acknowledgments

Figures (3)

  • Figure 1: Illustration of proposed DLD framework. $\mathcal{M}_\text{ref}$ uses all encoder layers to supervise the embeddings of $\mathcal{M}_\text{DS}$.
  • Figure 2: Comparing dynamic versions WavLM (random dropping based) vs Wav2Vec2 (learnable masking based). Circle depicts the utilized parameters for different encoder sizes.
  • Figure 3: Evaluating generalization capability of our framework on spoken language understanding task using FSC dataset. Baseline Accuracy (with full model $\mathcal{M}_\text{ref}$ = 97.1%)