Paraformer-v2: An improved non-autoregressive transformer for noise-robust speech recognition
Keyu An, Zerui Li, Zhifu Gao, Shiliang Zhang
TL;DR
Paraformer-v2 tackles the inefficiency of autoregressive decoders by adopting a non-autoregressive transformer for speech recognition. It replaces the CIF-based token embedding module with a CTC-based embedding extractor, improving multilingual adaptability and robustness to noise, and trains with a combined objective that includes CrossEntropy and CTCLoss. The approach uses a Viterbi-aligned CTC compression to align decoder inputs and achieves competitive accuracy with autoregressive conformers while delivering substantial speedups (over 20x) in decoding. Empirical results on AISHELL-1, LibriSpeech, and a 50k-hour English dataset show state-of-the-art NAT performance and strong noise robustness, indicating practical viability for real-time, noisy ASR systems.
Abstract
Attention-based encoder-decoder, e.g. transformer and its variants, generates the output sequence in an autoregressive (AR) manner. Despite its superior performance, AR model is computationally inefficient as its generation requires as many iterations as the output length. In this paper, we propose Paraformer-v2, an improved version of Paraformer, for fast, accurate, and noise-robust non-autoregressive speech recognition. In Paraformer-v2, we use a CTC module to extract the token embeddings, as the alternative to the continuous integrate-and-fire module in Paraformer. Extensive experiments demonstrate that Paraformer-v2 outperforms Paraformer on multiple datasets, especially on the English datasets (over 14% improvement on WER), and is more robust in noisy environments.