Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Purification Before Fusion: Toward Mask-Free Speech Enhancement for Robust Audio-Visual Speech Recognition

Linzhi Wu, Xingyu Zhang, Hao Yuan, Yakun Zhang, Changyan Zheng, Liang Xie, Tiejun Liu, Erwei Yin

TL;DR

The paper tackles robustness in audio-visual speech recognition under high noise by proposing a mask-free, purify-then-fuse framework that integrates speech enhancement with a bottleneck Conformer. It introduces an audio-visual bottleneck Conformer (AVBC) with a small set of learnable bottleneck tokens to implicitly purify audio features before fusion, and trains this with a joint AVSR and enhancement objective. Speech enhancement leverages reconstruction and perceptual losses to produce semantically rich audio representations, while the fusion stage focuses on integrating purified audio with visual cues via a Conformer encoder and a joint CTC/attention decoder. Evaluated on the LRS3 benchmark, the approach outperforms mask-based baselines across noisy conditions and demonstrates robustness to varied input scenarios, indicating practical benefits for real-world AVSR systems.

Abstract

Audio-visual speech recognition (AVSR) typically improves recognition accuracy in noisy environments by integrating noise-immune visual cues with audio signals. Nevertheless, high-noise audio inputs are prone to introducing adverse interference into the feature fusion process. To mitigate this, recent AVSR methods often adopt mask-based strategies to filter audio noise during feature interaction and fusion, yet such methods risk discarding semantically relevant information alongside noise. In this work, we propose an end-to-end noise-robust AVSR framework coupled with speech enhancement, eliminating the need for explicit noise mask generation. This framework leverages a Conformer-based bottleneck fusion module to implicitly refine noisy audio features with video assistance. By reducing modality redundancy and enhancing inter-modal interactions, our method preserves speech semantic integrity to achieve robust recognition performance. Experimental evaluations on the public LRS3 benchmark suggest that our method outperforms prior advanced mask-based baselines under noisy conditions.

Purification Before Fusion: Toward Mask-Free Speech Enhancement for Robust Audio-Visual Speech Recognition

TL;DR

The paper tackles robustness in audio-visual speech recognition under high noise by proposing a mask-free, purify-then-fuse framework that integrates speech enhancement with a bottleneck Conformer. It introduces an audio-visual bottleneck Conformer (AVBC) with a small set of learnable bottleneck tokens to implicitly purify audio features before fusion, and trains this with a joint AVSR and enhancement objective. Speech enhancement leverages reconstruction and perceptual losses to produce semantically rich audio representations, while the fusion stage focuses on integrating purified audio with visual cues via a Conformer encoder and a joint CTC/attention decoder. Evaluated on the LRS3 benchmark, the approach outperforms mask-based baselines across noisy conditions and demonstrates robustness to varied input scenarios, indicating practical benefits for real-world AVSR systems.

Abstract

Audio-visual speech recognition (AVSR) typically improves recognition accuracy in noisy environments by integrating noise-immune visual cues with audio signals. Nevertheless, high-noise audio inputs are prone to introducing adverse interference into the feature fusion process. To mitigate this, recent AVSR methods often adopt mask-based strategies to filter audio noise during feature interaction and fusion, yet such methods risk discarding semantically relevant information alongside noise. In this work, we propose an end-to-end noise-robust AVSR framework coupled with speech enhancement, eliminating the need for explicit noise mask generation. This framework leverages a Conformer-based bottleneck fusion module to implicitly refine noisy audio features with video assistance. By reducing modality redundancy and enhancing inter-modal interactions, our method preserves speech semantic integrity to achieve robust recognition performance. Experimental evaluations on the public LRS3 benchmark suggest that our method outperforms prior advanced mask-based baselines under noisy conditions.
Paper Structure (15 sections, 6 equations, 2 figures, 3 tables)

This paper contains 15 sections, 6 equations, 2 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Proposed Method
  3. Feature Extraction
  4. Audio-Visual Bottleneck Conformer
  5. Speech Feature Enhancement
  6. Fusion and Speech Recognition
  7. Experiments
  8. Data Processing
  9. Experimental Settings
  10. Results and Analysis
  11. Influence of the number of bottleneck tokens
  12. Effectiveness of the speech enhancement
  13. Noise robustness evaluation
  14. Experiments on various input conditions
  15. Conclusion

Figures (2)

  • Figure 1: Overall architecture of the proposed model.
  • Figure 2: Performance comparison of different token numbers at the first layer of our bottleneck Conformer under -5dB SNR babble noise. "w/o enh" indicates the model without speech enhancement.