Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Fast Streaming Transducer ASR Prototyping via Knowledge Distillation with Whisper

Iuliia Thorbecke, Juan Zuluaga-Gomez, Esaú Villatoro-Tello, Shashi Kumar, Pradeep Rangappa, Sergio Burdisso, Petr Motlicek, Karthik Pandia, Aravind Ganapathiraju

TL;DR

This work demonstrates that streaming Transformer-Transducer models can be trained from scratch in consumer and accessible GPUs in their entirety with pseudo-labeled speech from foundational speech models (FSM), and validate the proposed framework on 6 languages from CommonVoice and propose multiple heuristics to filter out hallucinated PLs.

Abstract

The training of automatic speech recognition (ASR) with little to no supervised data remains an open question. In this work, we demonstrate that streaming Transformer-Transducer (TT) models can be trained from scratch in consumer and accessible GPUs in their entirety with pseudo-labeled (PL) speech from foundational speech models (FSM). This allows training a robust ASR model just in one stage and does not require large data and computational budget compared to the two-step scenario with pre-training and fine-tuning. We perform a comprehensive ablation on different aspects of PL-based streaming TT models such as the impact of (1) shallow fusion of n-gram LMs, (2) contextual biasing with named entities, (3) chunk-wise decoding for low-latency streaming applications, and (4) TT overall performance as the function of the FSM size. Our results demonstrate that TT can be trained from scratch without supervised data, even with very noisy PLs. We validate the proposed framework on 6 languages from CommonVoice and propose multiple heuristics to filter out hallucinated PLs.

Fast Streaming Transducer ASR Prototyping via Knowledge Distillation with Whisper

TL;DR

This work demonstrates that streaming Transformer-Transducer models can be trained from scratch in consumer and accessible GPUs in their entirety with pseudo-labeled speech from foundational speech models (FSM), and validate the proposed framework on 6 languages from CommonVoice and propose multiple heuristics to filter out hallucinated PLs.

Abstract

The training of automatic speech recognition (ASR) with little to no supervised data remains an open question. In this work, we demonstrate that streaming Transformer-Transducer (TT) models can be trained from scratch in consumer and accessible GPUs in their entirety with pseudo-labeled (PL) speech from foundational speech models (FSM). This allows training a robust ASR model just in one stage and does not require large data and computational budget compared to the two-step scenario with pre-training and fine-tuning. We perform a comprehensive ablation on different aspects of PL-based streaming TT models such as the impact of (1) shallow fusion of n-gram LMs, (2) contextual biasing with named entities, (3) chunk-wise decoding for low-latency streaming applications, and (4) TT overall performance as the function of the FSM size. Our results demonstrate that TT can be trained from scratch without supervised data, even with very noisy PLs. We validate the proposed framework on 6 languages from CommonVoice and propose multiple heuristics to filter out hallucinated PLs.
Paper Structure (42 sections, 3 equations, 5 figures, 7 tables)

This paper contains 42 sections, 3 equations, 5 figures, 7 tables.

Table of Contents

  1. Introduction
  2. Contributions:
  3. Related Work
  4. From Encoder-Decoder to Transducer models
  5. Pseudo-labeling in ASR
  6. Knowledge Distillation with Large Models
  7. Experimental Setup
  8. Pseudo Labeling with Whisper
  9. Decoding with WhisperX pipeline
  10. Data filtering heuristics
  11. Transformer-Transducer Training
  12. Regularization with supervised data
  13. Enabling streaming decoding with multi-chunk training
  14. Language Modeling and Contextual Biasing
  15. Shallow fusion with Aho-Corasick
  16. ...and 27 more sections

Figures (5)

  • Figure 1: Proposed framework for efficient and fast streaming ASR prototyping with pseudo-labeled data. Transducer models are further improved via shallow fusion of n-gram LMs and contextual biasing of target named entities.
  • Figure 2: WERs for offline Zipformer models on six languages of CommonVoice. Models are trained with pseudo-labels from different Whisper model sizes (blue graphs). Adding 100h of supervised data during training (red graph) regularizes the training up to models with 700M params, especially for languages with less data.
  • Figure 3: Box plots of WERs for six languages of CommonVoice. Streaming Zipformer models are trained from scratch, with only PLs generated with different Whisper model sizes. Each box denotes 13 decoding configurations, ranging from challenging (320ms chunk with limited left context) to more relaxed (2560ms chunk with full left context) streaming settings. (Note different WER scaling on the y-axis.)
  • Figure 4: Ablations on WERs of Zipformer models for 6 languages of CommonVoice. We study the impact of mixing supervised data during training with pseudo-labeled of very low quality, i.e., Whisper-tiny.
  • Figure 5: WERs on the test set with different Whisper model configurations and chunk sizes of the VAD model.