SAV-SE: Scene-aware Audio-Visual Speech Enhancement with Selective State Space Model

TL;DR

This work addresses robust speech enhancement in noisy environments by exploiting environmental visual context through the SAV-SE task. The authors propose VSE, a Conformer–Mamba–ConMamba–based framework that fuses spectral audio features with scenario-aware audio and environmental visual embeddings to estimate a phase-sensitive mask $\mathbf{M}$ that yields the enhanced waveform. Extensive experiments on MUSIC, AVSpeech, and AudioSet show that VSE achieves state-of-the-art PESQ and STOI scores, with ablations confirming the contribution of both the scenario-aware audio and visualembeddings; Grad-CAM analyses reveal the model’s focus on sound-producing objects. The results demonstrate the practical value of environmental visual cues for robust AVSE, especially when facial cues are unavailable, and suggest future work on speech separation and multimodal robustness under partial data loss.

Abstract

Speech enhancement plays an essential role in various applications, and the integration of visual information has been demonstrated to bring substantial advantages. However, the majority of current research concentrates on the examination of facial and lip movements, which can be compromised or entirely inaccessible in scenarios where occlusions occur or when the camera view is distant. Whereas contextual visual cues from the surrounding environment have been overlooked: for example, when we see a dog bark, our brain has the innate ability to discern and filter out the barking noise. To this end, in this paper, we introduce a novel task, i.e. SAV-SE. To our best knowledge, this is the first proposal to use rich contextual information from synchronized video as auxiliary cues to indicate the type of noise, which eventually improves the speech enhancement performance. Specifically, we propose the VC-S$^2$E method, which incorporates the Conformer and Mamba modules for their complementary strengths. Extensive experiments are conducted on public MUSIC, AVSpeech and AudioSet datasets, where the results demonstrate the superiority of VC-S$^2$E over other competitive methods. We will make the source code publicly available. Project demo page: https://AVSEPage.github.io/

Figures (7)

• Figure 1: Our proposed SAVSE task where outputs from the audio and visual encoder are fused to refine and generate the enhanced audio. By incorporating the visual context from noise environments, it significantly enhances speech quality, particularly in situations where traditional audio-only techniques falter.
• Figure 2: The block diagram of our proposed VSE network which enhances human speech given the auxiliary visual cues from the noise scenario.
• Figure 3: The illustrations of (a) the BiMamba layer and (b) the Mamba block.
• Figure 4: The (a) PESQ and (b) STOI scores for various audio categories when using AudioSet as the noise dataset. The results are averaged over all of the SNR conditions.
• Figure 5: The Grad-CAM visualization of the focal areas in various video frames processed by our VSE model. Top row: original video frames; bottom row: corresponding Grad-CAM heatmaps which emphasize regions crucial for detecting noise sources, as identified by the video encoder. The color varying from blue to red indicates the more significant regions.