Joint Beam Search Integrating CTC, Attention, and Transducer Decoders

TL;DR

Experimental results demonstrate that the jointly trained 4D model outperforms the E2E-ASR models trained with only one individual decoder, and the proposed joint beam search algorithm outperforms the previously proposed CTC/attention decoding.

Abstract

End-to-end automatic speech recognition (E2E-ASR) can be classified by its decoder architectures, such as connectionist temporal classification (CTC), recurrent neural network transducer (RNN-T), attention-based encoder-decoder, and Mask-CTC models. Each decoder architecture has advantages and disadvantages, leading practitioners to switch between these different models depending on application requirements. Instead of building separate models, we propose a joint modeling scheme where four decoders (CTC, RNN-T, attention, and Mask-CTC) share the same encoder -- we refer to this as 4D modeling. The 4D model is trained jointly, which will bring model regularization and maximize the model robustness thanks to their complementary properties. To efficiently train the 4D model, we introduce a two-stage training strategy that stabilizes the joint training. In addition, we propose three novel joint beam search algorithms by combining three decoders (CTC, RNN-T, and attention) to further improve performance. These three beam search algorithms differ in which decoder is used as the primary decoder. We carefully evaluate the performance and computational tradeoffs associated with each algorithm. Experimental results demonstrate that the jointly trained 4D model outperforms the E2E-ASR models trained with only one individual decoder. Furthermore, we demonstrate that the proposed joint beam search algorithm outperforms the previously proposed CTC/attention decoding.

Figures (16)

• Figure 1: Alignment paths of CTC and RNN-T. The CTC alignment path is of length $T$, while the RNN-T alignment path is of length $(T + S)$.
• Figure 2: RNN-T prefix scoring (lines 3 and 4 in Algorithm 2).
• Figure 3: Normalized validation curves of the first, second training stages, and alternative trial ($\lambda_{\text{ctc}}$, $\lambda_{\text{rnnt}}$, $\lambda_{\text{att}}$, $\lambda_{\text{mlm}}$)
• Figure 4: Relationship between RTFs and WERs. The black, blue, and green dots represent the baselines, the 4D model without beam search, and the 4D model with beam search, respectively.
• Figure 5: Relationship between main beam size and RTF/WER. The black dots represent the baselines, while the green dots indicate the 4D model with RNN-T-driven joint beam search.