Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Keep Decoding Parallel with Effective Knowledge Distillation from Language Models to End-to-end Speech Recognisers

Michael Hentschel, Yuta Nishikawa, Tatsuya Komatsu, Yusuke Fujita

TL;DR

This work tackles injecting external language model knowledge into a non-autoregressive CTC-ASR without sacrificing parallel decoding. It introduces interAED-KD, distilling BERT token probabilities into both final and intermediate encoder layers through auxiliary attention decoders, while keeping the ASR model size constant at inference. Experiments on LibriSpeech show consistent WER improvements over baselines, with the best configuration achieving substantial reductions and maintaining fast greedy decoding; beam search with an external LM still offers gains but at higher latency. Overall, intermediate-layer distillation emerges as a key mechanism for effectively transferring LM information into ASR, offering a practical path to improved accuracy without decoding-time fusion costs.

Abstract

This study presents a novel approach for knowledge distillation (KD) from a BERT teacher model to an automatic speech recognition (ASR) model using intermediate layers. To distil the teacher's knowledge, we use an attention decoder that learns from BERT's token probabilities. Our method shows that language model (LM) information can be more effectively distilled into an ASR model using both the intermediate layers and the final layer. By using the intermediate layers as distillation target, we can more effectively distil LM knowledge into the lower network layers. Using our method, we achieve better recognition accuracy than with shallow fusion of an external LM, allowing us to maintain fast parallel decoding. Experiments on the LibriSpeech dataset demonstrate the effectiveness of our approach in enhancing greedy decoding with connectionist temporal classification (CTC).

Keep Decoding Parallel with Effective Knowledge Distillation from Language Models to End-to-end Speech Recognisers

TL;DR

This work tackles injecting external language model knowledge into a non-autoregressive CTC-ASR without sacrificing parallel decoding. It introduces interAED-KD, distilling BERT token probabilities into both final and intermediate encoder layers through auxiliary attention decoders, while keeping the ASR model size constant at inference. Experiments on LibriSpeech show consistent WER improvements over baselines, with the best configuration achieving substantial reductions and maintaining fast greedy decoding; beam search with an external LM still offers gains but at higher latency. Overall, intermediate-layer distillation emerges as a key mechanism for effectively transferring LM information into ASR, offering a practical path to improved accuracy without decoding-time fusion costs.

Abstract

This study presents a novel approach for knowledge distillation (KD) from a BERT teacher model to an automatic speech recognition (ASR) model using intermediate layers. To distil the teacher's knowledge, we use an attention decoder that learns from BERT's token probabilities. Our method shows that language model (LM) information can be more effectively distilled into an ASR model using both the intermediate layers and the final layer. By using the intermediate layers as distillation target, we can more effectively distil LM knowledge into the lower network layers. Using our method, we achieve better recognition accuracy than with shallow fusion of an external LM, allowing us to maintain fast parallel decoding. Experiments on the LibriSpeech dataset demonstrate the effectiveness of our approach in enhancing greedy decoding with connectionist temporal classification (CTC).
Paper Structure (14 sections, 12 equations, 1 figure, 3 tables)

This paper contains 14 sections, 12 equations, 1 figure, 3 tables.

Table of Contents

  1. Introduction
  2. CTC Acoustic Model
  3. Connectionist Temporal Classification
  4. Conformer Acoustic Model
  5. External Language Model
  6. Distillation into Intermediate Layers
  7. Experiments
  8. Dataset and Model Configuration
  9. Greedy Decoding
  10. LM Beam Search Decoding
  11. Improvement by Intermediate KD
  12. Decoding Samples
  13. Discussion
  14. Conclusion and Outlook

Figures (1)

  • Figure 1: Proposed model architecture with intermediate CTC loss and intermediate attention decoder for knowledge distillation. BERT's parameters are frozen during training the ASR model. The CTC decoders and the attention decoders share their parameters.