Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Semi-Autoregressive Streaming ASR With Label Context

Siddhant Arora, George Saon, Shinji Watanabe, Brian Kingsbury

TL;DR

This work tackles the latency-accuracy trade-off in streaming ASR by introducing a semi-autoregressive (SAR) model that injects label-context from previous blocks through a pretrained LM subnetwork. The approach retains intra-block non-autoregressive decoding while preserving autoregressive dependencies across blocks, aided by a novel alignment-decoding scheme and training with forced alignments. Empirical results across Tedlium-2, Librispeech-100, and Switchboard show substantial relative WER gains over streaming NAR baselines and competitive performance with streaming AR, all with about 2.5x lower latency; external text pretraining of the LM further improves accuracy. This method enables more practical streaming ASR by combining fast blockwise decoding with effective use of past predictions as context, and points toward scaling the LM and text-injection strategies for further gains.

Abstract

Non-autoregressive (NAR) modeling has gained significant interest in speech processing since these models achieve dramatically lower inference time than autoregressive (AR) models while also achieving good transcription accuracy. Since NAR automatic speech recognition (ASR) models must wait for the completion of the entire utterance before processing, some works explore streaming NAR models based on blockwise attention for low-latency applications. However, streaming NAR models significantly lag in accuracy compared to streaming AR and non-streaming NAR models. To address this, we propose a streaming "semi-autoregressive" ASR model that incorporates the labels emitted in previous blocks as additional context using a Language Model (LM) subnetwork. We also introduce a novel greedy decoding algorithm that addresses insertion and deletion errors near block boundaries while not significantly increasing the inference time. Experiments show that our method outperforms the existing streaming NAR model by 19% relative on Tedlium2, 16%/8% on Librispeech-100 clean/other test sets, and 19%/8% on the Switchboard(SWB)/Callhome(CH) test sets. It also reduced the accuracy gap with streaming AR and non-streaming NAR models while achieving 2.5x lower latency. We also demonstrate that our approach can effectively utilize external text data to pre-train the LM subnetwork to further improve streaming ASR accuracy.

Semi-Autoregressive Streaming ASR With Label Context

TL;DR

This work tackles the latency-accuracy trade-off in streaming ASR by introducing a semi-autoregressive (SAR) model that injects label-context from previous blocks through a pretrained LM subnetwork. The approach retains intra-block non-autoregressive decoding while preserving autoregressive dependencies across blocks, aided by a novel alignment-decoding scheme and training with forced alignments. Empirical results across Tedlium-2, Librispeech-100, and Switchboard show substantial relative WER gains over streaming NAR baselines and competitive performance with streaming AR, all with about 2.5x lower latency; external text pretraining of the LM further improves accuracy. This method enables more practical streaming ASR by combining fast blockwise decoding with effective use of past predictions as context, and points toward scaling the LM and text-injection strategies for further gains.

Abstract

Non-autoregressive (NAR) modeling has gained significant interest in speech processing since these models achieve dramatically lower inference time than autoregressive (AR) models while also achieving good transcription accuracy. Since NAR automatic speech recognition (ASR) models must wait for the completion of the entire utterance before processing, some works explore streaming NAR models based on blockwise attention for low-latency applications. However, streaming NAR models significantly lag in accuracy compared to streaming AR and non-streaming NAR models. To address this, we propose a streaming "semi-autoregressive" ASR model that incorporates the labels emitted in previous blocks as additional context using a Language Model (LM) subnetwork. We also introduce a novel greedy decoding algorithm that addresses insertion and deletion errors near block boundaries while not significantly increasing the inference time. Experiments show that our method outperforms the existing streaming NAR model by 19% relative on Tedlium2, 16%/8% on Librispeech-100 clean/other test sets, and 19%/8% on the Switchboard(SWB)/Callhome(CH) test sets. It also reduced the accuracy gap with streaming AR and non-streaming NAR models while achieving 2.5x lower latency. We also demonstrate that our approach can effectively utilize external text data to pre-train the LM subnetwork to further improve streaming ASR accuracy.
Paper Structure (12 sections, 6 equations, 1 figure, 3 tables, 1 algorithm)

This paper contains 12 sections, 6 equations, 1 figure, 3 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Problem Formulation
  3. Method
  4. Training and Inference
  5. Experiments
  6. Datasets
  7. Baseline
  8. Experimental Setups
  9. Results and Discussion
  10. Conclusion
  11. Acknowledgements
  12. References

Figures (1)

  • Figure 1: Schematics of our semi-autoregressive streaming ASR model