DGFNet: End-to-End Audio-Visual Source Separation Based on Dynamic Gating Fusion

TL;DR

DGFNet tackles the cocktail-party problem by dynamically fusing audio and visual cues at the encoder bottleneck and enhancing audio representations with an attention module. The approach couples a Dynamic Gating Fusion Module with an Audio-Visual Transformer to enable data-dependent, cross-modal interactions that improve spectrogram masking. Empirical results on MUSIC and MUSIC-21 show consistent gains in SDR, SIR, and SAR over strong baselines, with ablations confirming the efficacy of bottleneck fusion and the gating mechanism. The method demonstrates strong performance in multi-source and noisy environments, highlighting its practical potential for end-to-end AV source separation.

Abstract

Current Audio-Visual Source Separation methods primarily adopt two design strategies. The first strategy involves fusing audio and visual features at the bottleneck layer of the encoder, followed by processing the fused features through the decoder. However, when there is a significant disparity between the two modalities, this approach may lead to the loss of critical information. The second strategy avoids direct fusion and instead relies on the decoder to handle the interaction between audio and visual features. Nonetheless, if the encoder fails to integrate information across modalities adequately, the decoder may be unable to effectively capture the complex relationships between them. To address these issues, this paper proposes a dynamic fusion method based on a gating mechanism that dynamically adjusts the modality fusion degree. This approach mitigates the limitations of solely relying on the decoder and facilitates efficient collaboration between audio and visual features. Additionally, an audio attention module is introduced to enhance the expressive capacity of audio features, thereby further improving model performance. Experimental results demonstrate that our method achieves significant performance improvements on two benchmark datasets, validating its effectiveness and advantages in Audio-Visual Source Separation tasks.

Figures (6)

• Figure 1: The overall architecture of the DGFNet. The model comprises the Audio-Visual feature extraction network, Audio-Visual feature fusion network, and query-based Audio-Visual Transformer network. The feature extraction network extracts object and motion features from video segments $V_{k}$ using an object detector, motion encoder, and image encoder. The audio feature extraction network extracts features from the mixed audio signal $s_{mix}(t)$. The Audio-Visual feature fusion network integrates visual and audio features through the Dynamic Gating Fusion Module (DGFM). Finally, the Audio-Visual Transformer network generates mask embeddings using the query mechanism for target audio separation.
• Figure 2: Audio Attention Module.
• Figure 3: Dynamic Gating Fusion Module.
• Figure 4: Qualitative results on MUSIC test set.
• Figure 5: Distribution of Weight Coefficient Mean.