Robust Self-Supervised Audio-Visual Speech Recognition
Bowen Shi, Wei-Ning Hsu, Abdelrahman Mohamed
TL;DR
This work advances robust audio-visual speech recognition by adopting a self-supervised AV-HuBERT framework enhanced with noise-augmented pretraining and finetuning. It demonstrates substantial, data-efficient gains on the LRS3 benchmark, achieving ~50% relative WER reductions with under 10% of labeled data and strong robustness across diverse noisy environments. The results show that multimodal representations and deliberate exposure to noise during pretraining and finetuning enable AVSR to outperform audio-only baselines by large margins, particularly in challenging conditions such as overlapping speech. These findings highlight the practical impact of self-supervised multimodal learning for low-resource and multilingual speech recognition tasks.
Abstract
Audio-based automatic speech recognition (ASR) degrades significantly in noisy environments and is particularly vulnerable to interfering speech, as the model cannot determine which speaker to transcribe. Audio-visual speech recognition (AVSR) systems improve robustness by complementing the audio stream with the visual information that is invariant to noise and helps the model focus on the desired speaker. However, previous AVSR work focused solely on the supervised learning setup; hence the progress was hindered by the amount of labeled data available. In this work, we present a self-supervised AVSR framework built upon Audio-Visual HuBERT (AV-HuBERT), a state-of-the-art audio-visual speech representation learning model. On the largest available AVSR benchmark dataset LRS3, our approach outperforms prior state-of-the-art by ~50% (28.0% vs. 14.1%) using less than 10% of labeled data (433hr vs. 30hr) in the presence of babble noise, while reducing the WER of an audio-based model by over 75% (25.8% vs. 5.8%) on average.