Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

EffectiveASR: A Single-Step Non-Autoregressive Mandarin Speech Recognition Architecture with High Accuracy and Inference Speed

Ziyang Zhuang, Chenfeng Miao, Kun Zou, Ming Fang, Tao Wei, Zijian Li, Ning Cheng, Wei Hu, Shaojun Wang, Jing Xiao

TL;DR

EffectiveASR addresses the speed-accuracy trade-off in Mandarin ASR by introducing a single-step non-autoregressive architecture that uses an IMV-based alignment generator during training and an alignment predictor during inference to produce monotonic alignments. It also includes distance-aware attention reconstruction to map acoustic frames to output tokens, and trains end-to-end with cross-entropy plus an alignment loss. The method achieves competitive CERs on AISHELL-1 and AISHELL-2, including 4.26%/4.62% on AISHELL-1 and 5.76% on AISHELL-2, while delivering approximately 30× faster decoding than a strong AR Conformer. These results demonstrate a compact, efficient NAR model with practical inference speedups and strong accuracy, with potential applicability beyond Mandarin and toward English in future work.

Abstract

Non-autoregressive (NAR) automatic speech recognition (ASR) models predict tokens independently and simultaneously, bringing high inference speed. However, there is still a gap in the accuracy of the NAR models compared to the autoregressive (AR) models. In this paper, we propose a single-step NAR ASR architecture with high accuracy and inference speed, called EffectiveASR. It uses an Index Mapping Vector (IMV) based alignment generator to generate alignments during training, and an alignment predictor to learn the alignments for inference. It can be trained end-to-end (E2E) with cross-entropy loss combined with alignment loss. The proposed EffectiveASR achieves competitive results on the AISHELL-1 and AISHELL-2 Mandarin benchmarks compared to the leading models. Specifically, it achieves character error rates (CER) of 4.26%/4.62% on the AISHELL-1 dev/test dataset, which outperforms the AR Conformer with about 30x inference speedup.

EffectiveASR: A Single-Step Non-Autoregressive Mandarin Speech Recognition Architecture with High Accuracy and Inference Speed

TL;DR

EffectiveASR addresses the speed-accuracy trade-off in Mandarin ASR by introducing a single-step non-autoregressive architecture that uses an IMV-based alignment generator during training and an alignment predictor during inference to produce monotonic alignments. It also includes distance-aware attention reconstruction to map acoustic frames to output tokens, and trains end-to-end with cross-entropy plus an alignment loss. The method achieves competitive CERs on AISHELL-1 and AISHELL-2, including 4.26%/4.62% on AISHELL-1 and 5.76% on AISHELL-2, while delivering approximately 30× faster decoding than a strong AR Conformer. These results demonstrate a compact, efficient NAR model with practical inference speedups and strong accuracy, with potential applicability beyond Mandarin and toward English in future work.

Abstract

Non-autoregressive (NAR) automatic speech recognition (ASR) models predict tokens independently and simultaneously, bringing high inference speed. However, there is still a gap in the accuracy of the NAR models compared to the autoregressive (AR) models. In this paper, we propose a single-step NAR ASR architecture with high accuracy and inference speed, called EffectiveASR. It uses an Index Mapping Vector (IMV) based alignment generator to generate alignments during training, and an alignment predictor to learn the alignments for inference. It can be trained end-to-end (E2E) with cross-entropy loss combined with alignment loss. The proposed EffectiveASR achieves competitive results on the AISHELL-1 and AISHELL-2 Mandarin benchmarks compared to the leading models. Specifically, it achieves character error rates (CER) of 4.26%/4.62% on the AISHELL-1 dev/test dataset, which outperforms the AR Conformer with about 30x inference speedup.
Paper Structure (10 sections, 8 equations, 3 figures, 1 table)

This paper contains 10 sections, 8 equations, 3 figures, 1 table.

Table of Contents

  1. Introduction
  2. Methods
  3. Overview
  4. Alignment generator
  5. Attention reconstruction
  6. Alignment predictor
  7. Experiments
  8. Experimental Setup
  9. Results
  10. Conclusion

Figures (3)

  • Figure 1: The model architecture of the proposed EffectiveASR. ($\bigotimes$ represents the matrix multiplication operation)
  • Figure 2: Visualization of alignment generator and attention reconstruction calculation process. (Operation $\Delta$ is defined as Eq. (\ref{['delta_e']}). Operation Cum-Sum is defined as Eq. (\ref{['sum_e']}).)
  • Figure 3: Alignment plots of both the baseline and proposed model. The first plot is the alignment plot of the Conformer, while the second plot is the alignment constructed by the proposed EffectiveASR. The horizontal axis represents the input frame step, and the vertical axis represents the output step.