Uncovering the Visual Contribution in Audio-Visual Speech Recognition

TL;DR

This paper addresses whether AVSR systems truly exploit visual information by benchmarking against human speech perception and using effective SNR gains at $0~dB$ to quantify visual contribution. It analyzes three SOTA AVSR models—Auto-AVSR, AVEC, and AV-RelScore—on LRS2 and LRS3, across SNR conditions, occlusion scenarios, and MaFI-based word informativeness. The findings show that high $WER$ does not imply a large visual contribution; AVEC achieves the largest effective SNR gains despite poorer $WER$, suggesting current methods underutilize visual cues. The authors advocate reporting effective SNR gains alongside $WER$ and call for design strategies that better leverage visual information in noisy conditions, aligning system evaluation more closely with human perception.

Abstract

Audio-Visual Speech Recognition (AVSR) combines auditory and visual speech cues to enhance the accuracy and robustness of speech recognition systems. Recent advancements in AVSR have improved performance in noisy environments compared to audio-only counterparts. However, the true extent of the visual contribution, and whether AVSR systems fully exploit the available cues in the visual domain, remains unclear. This paper assesses AVSR systems from a different perspective, by considering human speech perception. We use three systems: Auto-AVSR, AVEC and AV-RelScore. We first quantify the visual contribution using effective SNR gains at 0 dB and then investigate the use of visual information in terms of its temporal distribution and word-level informativeness. We show that low WER does not guarantee high SNR gains. Our results suggest that current methods do not fully exploit visual information, and we recommend future research to report effective SNR gains alongside WERs.

Figures (2)

• Figure 1: Performance of six models (three methods, both audio-only and audio-visual) on LRS2 and LRS3 test sets under different levels of pink noise.
• Figure 2: Examples of video frames before and after inpainting occlusion.