Audio-Visual Separation with Hierarchical Fusion and Representation Alignment

TL;DR

This work investigates how acoustic properties influence fusion strategies for self-supervised audio-visual separation and introduces a hierarchical fusion framework that combines middle and late fusion. It further enhances semantic understanding through audio representation alignment by matching the U-Net encoder features to pre-trained audio embeddings (CLAP), guided by a proposed alignment loss. The method achieves state-of-the-art SDR across MUSIC, MUSIC-21, and VGGSound, and analyses show reduced modality gap and richer semantic audio representations. The approach demonstrates robust performance across diverse sound types and suggests future work in self-supervised visual localisation to further improve separation quality.

Abstract

Self-supervised audio-visual source separation leverages natural correlations between audio and vision modalities to separate mixed audio signals. In this work, we first systematically analyse the performance of existing multimodal fusion methods for audio-visual separation task, demonstrating that the performance of different fusion strategies is closely linked to the characteristics of the sound: middle fusion is better suited for handling short, transient sounds, while late fusion is more effective for capturing sustained and harmonically rich sounds. We thus propose a hierarchical fusion strategy that effectively integrates both fusion stages. In addition, training can be made easier by incorporating high-quality external audio representations, rather than relying solely on the audio branch to learn them independently. To explore this, we propose a representation alignment approach that aligns the latent features of the audio encoder with embeddings extracted from pre-trained audio models. Extensive experiments on MUSIC, MUSIC-21 and VGGSound datasets demonstrate that our approach achieves state-of-the-art results, surpassing existing methods under the self-supervised setting. We further analyse the impact of representation alignment on audio features, showing that it reduces modality gap between the audio and visual modalities.

Figures (5)

• Figure 1: Relationship Between Acoustic Properties of Musical Instruments and Fusion Strategies. Instruments with shorter transient properties and simpler harmonic structures are more suited to middle fusion. Conversely, instruments with sustained notes and complex harmonic structures benefit more from late fusion. Details can be found in Appendix \ref{['app:s1']}.
• Figure 2: Pipeline of our proposed method. The pipeline consists of three key components: audio-visual feature extraction, hierarchical fusion, and representation alignment. It takes an audio mixture and corresponding video frames as input. The Audio-Visual Feature Extraction module processes the input through dedicated encoders, extracting audio features from spectrograms and CLAP and extracting visual features from CLIP. Hierarchical fusion includes middle fusion and late fusion, which happens at the bottleneck of the audio U-Net and the final layer of the audio decoder separately.
• Figure 3: Qualitative Performance on MUSIC dataset. We compared our method (the fourth row) with dong2023clipsep (the third row). More results in Appendix \ref{['app:more_examples']}.
• Figure A1: More Visualisation Examples on MUSIC dataset. We visualise the spectrograms for original audio, mixed audio, and predictions from different audio separation models with corresponding source query frames. We compared our method (the fourth column) with Clipsep (the third column).
• Figure S2: More Visualisation Examples on MUSIC dataset. We visualise the spectrograms for original audio, mixed audio, and predictions from different audio separation models with corresponding source query frames. We compared our method (the fourth column) with Clipsep (the third column).