Bridging The Multi-Modality Gaps of Audio, Visual and Linguistic for Speech Enhancement

TL;DR

This paper tackles robust speech enhancement by bridging gaps across audio, visual, and linguistic modalities. It introduces DLAV-SE, a diffusion-model–based AVSE framework that fuses lip-based visual cues with a language-guided training pathway, using cross-modal knowledge transfer (CMKT) via two fusion strategies (MHCA and OT) and a pretrained language model during training only. Empirical results on Mandarin, English, and real-world driving datasets show that incorporating linguistic context substantially improves speech quality, intelligibility, and artifact reduction beyond state-of-the-art AVSE methods, while incurring no inference-time cost from the language model. The work demonstrates the practical viability of linguistically informed multimodal SE and highlights diffusion-based approaches as a promising direction for robust, real-time speech enhancement in challenging acoustic environments.

Abstract

Speech enhancement (SE) aims to improve the quality and intelligibility of speech in noisy environments. Recent studies have shown that incorporating visual cues in audio signal processing can enhance SE performance. Given that human speech communication naturally involves audio, visual, and linguistic modalities, it is reasonable to expect additional improvements by integrating linguistic information. However, effectively bridging these modality gaps, particularly during knowledge transfer remains a significant challenge. In this paper, we propose a novel multi-modal learning framework, termed DLAV-SE, which leverages a diffusion-based model integrating audio, visual, and linguistic information for audio-visual speech enhancement (AVSE). Within this framework, the linguistic modality is modeled using a pretrained language model (PLM), which transfers linguistic knowledge to the audio-visual domain through a cross-modal knowledge transfer (CMKT) mechanism during training. After training, the PLM is no longer required at inference, as its knowledge is embedded into the AVSE model through the CMKT process. We conduct a series of SE experiments to evaluate the effectiveness of our approach. Results show that the proposed DLAV-SE system significantly improves speech quality and reduces generative artifacts, such as phonetic confusion, compared to state-of-the-art (SOTA) methods. Furthermore, visualization analyses confirm that the CMKT method enhances the generation quality of the AVSE outputs. These findings highlight both the promise of diffusion-based methods for advancing AVSE and the value of incorporating linguistic information to further improve system performance.

Figures (3)

• Figure 1: The hybrid predictive–generative diffusion-based DLAV-SE system is trained with integrated linguistic information. The linguistic module, including the BERT model, cosine distance loss, and fusion layer, is utilized only during the training phase. During inference, only the components enclosed by the dotted line are activated.
• Figure 2: The inference process of the DLAV-SE System. After applying the short-time Fourier transform (STFT), the input noisy spectrogram is first passed through the predictive model to generate a preliminary denoised output. This intermediate spectrogram is then fed into the generative model, which performs reverse diffusion over T steps. Each diffusion step is implemented using a U-Net architecture UNet.
• Figure 3: Spectrograms of (a) noisy speech, (b) clean speech, (c) enhanced speech by our model "DLAV-SE (A)", (d) enhanced speech by our model "DLAV-SE (A+V)", (e) enhanced speech by our model "DLAV-SE (A+L)", (f) enhanced speech by our model "DLAV-SE (A+V+L)" for an example in the TMSV dataset. The vertical axis represents frequency, and the horizontal axis represents time.