Multilingual Audio-Visual Speech Recognition with Hybrid CTC/RNN-T Fast Conformer

TL;DR

This work tackles multilingual AVSR under challenging noisy conditions by extending the Fast Conformer to process audio and visual streams with a hybrid CTC/RNN-T loss, enabling end-to-end training. It leverages generated transcriptions from large multilingual datasets (VoxCeleb2 and AVSpeech) to scale data across six languages, achieving new state-of-the-art results on LRS3 (WER ~0.8% for AV) and a substantial average WER reduction on MuAViC (≈11.94% vs baseline). The model supports audio-only, visual-only, and audio-visual inference at test time using modality masking and dropout, highlighting robustness and flexibility. The approach is implemented with careful architectural choices (early fusion, 18 Conformer blocks for audio and visual streams, 8 blocks for AV encoder, and specific training details), and will be open-sourced through NVIDIA NeMo for reproducibility and broader adoption.

Abstract

Humans are adept at leveraging visual cues from lip movements for recognizing speech in adverse listening conditions. Audio-Visual Speech Recognition (AVSR) models follow similar approach to achieve robust speech recognition in noisy conditions. In this work, we present a multilingual AVSR model incorporating several enhancements to improve performance and audio noise robustness. Notably, we adapt the recently proposed Fast Conformer model to process both audio and visual modalities using a novel hybrid CTC/RNN-T architecture. We increase the amount of audio-visual training data for six distinct languages, generating automatic transcriptions of unlabelled multilingual datasets (VoxCeleb2 and AVSpeech). Our proposed model achieves new state-of-the-art performance on the LRS3 dataset, reaching WER of 0.8%. On the recently introduced MuAViC benchmark, our model yields an absolute average-WER reduction of 11.9% in comparison to the original baseline. Finally, we demonstrate the ability of the proposed model to perform audio-only, visual-only, and audio-visual speech recognition at test time.

Figures (1)

• Figure 1: Audio-Visual Fast Conformer architecture. The model is trained end-to-end using CTC and RNN-T losses and takes both audio waveforms and lip movements as inputs.