Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Paraformer: Fast and Accurate Parallel Transformer for Non-autoregressive End-to-End Speech Recognition

Zhifu Gao, Shiliang Zhang, Ian McLoughlin, Zhijie Yan

TL;DR

Paraformer tackles the inefficiency of autoregressive decoders in Transformer ASR by introducing CIF‑based token counting, a GLM‑driven sampler for inter‑token context, and MWER‑driven negative sampling. The architecture enables a single parallel pass during inference while training remains multi‑component, balancing token predictions with contextual dependencies. Across AISHELL‑1/2 and a large 20k‑hour industrial Mandarin task, Paraformer achieves AR‑level accuracy with about 10× faster inference, demonstrating strong scalability and practical impact for fast, accurate end‑to‑end speech recognition.

Abstract

Transformers have recently dominated the ASR field. Although able to yield good performance, they involve an autoregressive (AR) decoder to generate tokens one by one, which is computationally inefficient. To speed up inference, non-autoregressive (NAR) methods, e.g. single-step NAR, were designed, to enable parallel generation. However, due to an independence assumption within the output tokens, performance of single-step NAR is inferior to that of AR models, especially with a large-scale corpus. There are two challenges to improving single-step NAR: Firstly to accurately predict the number of output tokens and extract hidden variables; secondly, to enhance modeling of interdependence between output tokens. To tackle both challenges, we propose a fast and accurate parallel transformer, termed Paraformer. This utilizes a continuous integrate-and-fire based predictor to predict the number of tokens and generate hidden variables. A glancing language model (GLM) sampler then generates semantic embeddings to enhance the NAR decoder's ability to model context interdependence. Finally, we design a strategy to generate negative samples for minimum word error rate training to further improve performance. Experiments using the public AISHELL-1, AISHELL-2 benchmark, and an industrial-level 20,000 hour task demonstrate that the proposed Paraformer can attain comparable performance to the state-of-the-art AR transformer, with more than 10x speedup.

Paraformer: Fast and Accurate Parallel Transformer for Non-autoregressive End-to-End Speech Recognition

TL;DR

Paraformer tackles the inefficiency of autoregressive decoders in Transformer ASR by introducing CIF‑based token counting, a GLM‑driven sampler for inter‑token context, and MWER‑driven negative sampling. The architecture enables a single parallel pass during inference while training remains multi‑component, balancing token predictions with contextual dependencies. Across AISHELL‑1/2 and a large 20k‑hour industrial Mandarin task, Paraformer achieves AR‑level accuracy with about 10× faster inference, demonstrating strong scalability and practical impact for fast, accurate end‑to‑end speech recognition.

Abstract

Transformers have recently dominated the ASR field. Although able to yield good performance, they involve an autoregressive (AR) decoder to generate tokens one by one, which is computationally inefficient. To speed up inference, non-autoregressive (NAR) methods, e.g. single-step NAR, were designed, to enable parallel generation. However, due to an independence assumption within the output tokens, performance of single-step NAR is inferior to that of AR models, especially with a large-scale corpus. There are two challenges to improving single-step NAR: Firstly to accurately predict the number of output tokens and extract hidden variables; secondly, to enhance modeling of interdependence between output tokens. To tackle both challenges, we propose a fast and accurate parallel transformer, termed Paraformer. This utilizes a continuous integrate-and-fire based predictor to predict the number of tokens and generate hidden variables. A glancing language model (GLM) sampler then generates semantic embeddings to enhance the NAR decoder's ability to model context interdependence. Finally, we design a strategy to generate negative samples for minimum word error rate training to further improve performance. Experiments using the public AISHELL-1, AISHELL-2 benchmark, and an industrial-level 20,000 hour task demonstrate that the proposed Paraformer can attain comparable performance to the state-of-the-art AR transformer, with more than 10x speedup.
Paper Structure (12 sections, 7 equations, 3 figures, 3 tables)

This paper contains 12 sections, 7 equations, 3 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Methods
  3. Overview
  4. Predictor
  5. Sampler
  6. Loss Function
  7. Experiments
  8. Experimental Setup
  9. AISHELL-1 and AISHELL-2 task
  10. Industrial 20,000 hour task
  11. Discussion
  12. Conclusion

Figures (3)

  • Figure 1: Analysis of different error types for three systems, evaluated on the industrial 20,000 hour task.
  • Figure 2: Structure of the proposed Paraformer.
  • Figure 3: An illustration of the CIF process ($\mathbf{\beta}$ is set to 1).