Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Noise-Robust AV-ASR Using Visual Features Both in the Whisper Encoder and Decoder

Zhengyang Li, Thomas Graave, Björn Möller, Zehang Wu, Matthias Franz, Tim Fingscheidt

TL;DR

This work tackles noise-robust audiovisual automatic speech recognition (AV-ASR) by introducing a dual-use visual fusion that incorporates AV-HuBERT visual features into both the Whisper encoder and decoder, enabling learned audiovisual interactions and modality-aware decoding. The method yields consistent noise-robustness gains across Whisper model sizes and substantially outperforms Flamingo-based middle fusion under noisy conditions, achieving a new state-of-the-art on the LRS3 AV-ASR benchmark when fine-tuned on 1929 hours of audiovisual data. Key results include relative improvements at 0 dB babble (e.g., $WER$ reductions of $35\%$ to $57\%$) and average WERs of $4.08\%$ (MUSAN) and $4.43\%$ (NoiseX), highlighting practical gains in real-world noisy environments. The work provides open-source code to facilitate replication and further research in robust multimodal speech recognition.

Abstract

In audiovisual automatic speech recognition (AV-ASR) systems, information fusion of visual features in a pre-trained ASR has been proven as a promising method to improve noise robustness. In this work, based on the prominent Whisper ASR, first, we propose a simple and effective visual fusion method -- use of visual features both in encoder and decoder (dual-use) -- to learn the audiovisual interactions in the encoder and to weigh modalities in the decoder. Second, we compare visual fusion methods in Whisper models of various sizes. Our proposed dual-use method shows consistent noise robustness improvement, e.g., a 35% relative improvement (WER: 4.41% vs. 6.83%) based on Whisper small, and a 57% relative improvement (WER: 4.07% vs. 9.53%) based on Whisper medium, compared to typical reference middle fusion in babble noise with a signal-to-noise ratio (SNR) of 0dB. Third, we conduct ablation studies examining the impact of various module designs and fusion options. Fine-tuned on 1929 hours of audiovisual data, our dual-use method using Whisper medium achieves 4.08% (MUSAN babble noise) and 4.43% (NoiseX babble noise) average WER across various SNRs, thereby establishing a new state-of-the-art in noisy conditions on the LRS3 AV-ASR benchmark. Our code is at https://github.com/ifnspaml/Dual-Use-AVASR

Noise-Robust AV-ASR Using Visual Features Both in the Whisper Encoder and Decoder

TL;DR

This work tackles noise-robust audiovisual automatic speech recognition (AV-ASR) by introducing a dual-use visual fusion that incorporates AV-HuBERT visual features into both the Whisper encoder and decoder, enabling learned audiovisual interactions and modality-aware decoding. The method yields consistent noise-robustness gains across Whisper model sizes and substantially outperforms Flamingo-based middle fusion under noisy conditions, achieving a new state-of-the-art on the LRS3 AV-ASR benchmark when fine-tuned on 1929 hours of audiovisual data. Key results include relative improvements at 0 dB babble (e.g., reductions of to ) and average WERs of (MUSAN) and (NoiseX), highlighting practical gains in real-world noisy environments. The work provides open-source code to facilitate replication and further research in robust multimodal speech recognition.

Abstract

In audiovisual automatic speech recognition (AV-ASR) systems, information fusion of visual features in a pre-trained ASR has been proven as a promising method to improve noise robustness. In this work, based on the prominent Whisper ASR, first, we propose a simple and effective visual fusion method -- use of visual features both in encoder and decoder (dual-use) -- to learn the audiovisual interactions in the encoder and to weigh modalities in the decoder. Second, we compare visual fusion methods in Whisper models of various sizes. Our proposed dual-use method shows consistent noise robustness improvement, e.g., a 35% relative improvement (WER: 4.41% vs. 6.83%) based on Whisper small, and a 57% relative improvement (WER: 4.07% vs. 9.53%) based on Whisper medium, compared to typical reference middle fusion in babble noise with a signal-to-noise ratio (SNR) of 0dB. Third, we conduct ablation studies examining the impact of various module designs and fusion options. Fine-tuned on 1929 hours of audiovisual data, our dual-use method using Whisper medium achieves 4.08% (MUSAN babble noise) and 4.43% (NoiseX babble noise) average WER across various SNRs, thereby establishing a new state-of-the-art in noisy conditions on the LRS3 AV-ASR benchmark. Our code is at https://github.com/ifnspaml/Dual-Use-AVASR
Paper Structure (10 sections, 2 equations, 1 figure, 3 tables)

This paper contains 10 sections, 2 equations, 1 figure, 3 tables.

Table of Contents

  1. Introduction
  2. Methods
  3. Baseline Whisper ASR Based on Acoustics
  4. Proposed Dual Visual Features Use for AV-ASR
  5. Evaluation and Discussion
  6. Experimental Setup
  7. Results and Discussion
  8. Conclusions
  9. Acknowledgments
  10. Compliance with Ethical Standards

Figures (1)

  • Figure 1: The audiovisual speech recognition system with our proposed dual visual features use in both the Whisper encoder and decoder.