Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Cross-Modal Bottleneck Fusion For Noise Robust Audio-Visual Speech Recognition

Seaone Ok, Min Jun Choi, Eungbeom Kim, Seungu Han, Kyogu Lee

TL;DR

This work tackles noise-robust audio-visual speech recognition (AVSR) by introducing CoBRA, a cross-modal bottleneck fusion framework that mediates modality exchange through a compact set of learnable tokens. Audio and visual encoders interact exclusively via bottleneck tokens, enabling the audio stream to leverage visual cues even in adverse conditions; fusion depth and bottleneck size are shown to critically impact performance. Empirically, CoBRA achieves strong results on LRS3 and LRS2 with limited training data, including notable improvements under noisy conditions, and exhibits noise-adaptive visual reliance via attention rollout analyses. The approach offers data-efficient robustness for AVSR and suggests avenues for scaling with larger pretraining and additional modalities.

Abstract

Audio-Visual Speech Recognition (AVSR) leverages both acoustic and visual cues to improve speech recognition under noisy conditions. A central question is how to design a fusion mechanism that allows the model to effectively exploit visual information when the audio signal is degraded, while maintaining strong performance on clean speech. We propose CoBRA (Cross-modal Bottleneck for Robust AVSR), a bottleneck-based fusion framework that introduces a compact set of learnable tokens to mediate cross-modal exchange. By regulating information flow through these tokens, the audio stream can reliably access essential visual cues even under adverse or out-of-domain noise. Despite limited training data, our model surpasses comparable baselines and remains competitive with large-scale systems through noise-adaptive fusion, demonstrating both efficiency and robustness. Ablation studies highlight that the depth of fusion is the most critical factor, underscoring its importance in designing robust AVSR systems.

Cross-Modal Bottleneck Fusion For Noise Robust Audio-Visual Speech Recognition

TL;DR

This work tackles noise-robust audio-visual speech recognition (AVSR) by introducing CoBRA, a cross-modal bottleneck fusion framework that mediates modality exchange through a compact set of learnable tokens. Audio and visual encoders interact exclusively via bottleneck tokens, enabling the audio stream to leverage visual cues even in adverse conditions; fusion depth and bottleneck size are shown to critically impact performance. Empirically, CoBRA achieves strong results on LRS3 and LRS2 with limited training data, including notable improvements under noisy conditions, and exhibits noise-adaptive visual reliance via attention rollout analyses. The approach offers data-efficient robustness for AVSR and suggests avenues for scaling with larger pretraining and additional modalities.

Abstract

Audio-Visual Speech Recognition (AVSR) leverages both acoustic and visual cues to improve speech recognition under noisy conditions. A central question is how to design a fusion mechanism that allows the model to effectively exploit visual information when the audio signal is degraded, while maintaining strong performance on clean speech. We propose CoBRA (Cross-modal Bottleneck for Robust AVSR), a bottleneck-based fusion framework that introduces a compact set of learnable tokens to mediate cross-modal exchange. By regulating information flow through these tokens, the audio stream can reliably access essential visual cues even under adverse or out-of-domain noise. Despite limited training data, our model surpasses comparable baselines and remains competitive with large-scale systems through noise-adaptive fusion, demonstrating both efficiency and robustness. Ablation studies highlight that the depth of fusion is the most critical factor, underscoring its importance in designing robust AVSR systems.
Paper Structure (20 sections, 5 equations, 3 figures, 2 tables)

This paper contains 20 sections, 5 equations, 3 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Related Works
  3. End-to-End Conformer AVSR
  4. Fusion Methods in AVSR
  5. Attention Bottleneck Transformer
  6. Method
  7. Overall architecture
  8. Bottleneck-Based Modality Fusion
  9. Training Objective
  10. Experimental Setup
  11. Datasets
  12. Implementation Details
  13. Results
  14. Main Results
  15. Ablation studies
  16. ...and 5 more sections

Figures (3)

  • Figure 1: Overall architecture of the proposed model.
  • Figure 2: Illustration of bottleneck fusion strategies: (left) sequential fusion and (right) mean fusion.
  • Figure 3: Variation of normalized cross-modal influence under different SNR conditions