Semantic Grouping Network for Audio Source Separation

TL;DR

This work tackles audio source separation without relying on visual cues by introducing SGN, a network that directly disentangles per-source semantics from audio using learnable source class tokens and a category-aware grouping mechanism. The method extracts category-specific representations from the mixture spectrogram and uses a light reconstruction module to produce source masks, trained with a combination of cross-entropy and binary cross-entropy losses. SGN demonstrates strong performance across MUSIC, FUSS, MUSDB18, and VGG-Sound, surpassing many audio-only baselines and approaching or exceeding some audio-visual models that do not rely on extra visuals. The approach also supports a flexible number of sources and exhibits clear, interpretable category-aware embeddings, suggesting practical impact for robust audio separation in diverse environments.

Abstract

Recently, audio-visual separation approaches have taken advantage of the natural synchronization between the two modalities to boost audio source separation performance. They extracted high-level semantics from visual inputs as the guidance to help disentangle sound representation for individual sources. Can we directly learn to disentangle the individual semantics from the sound itself? The dilemma is that multiple sound sources are mixed together in the original space. To tackle the difficulty, in this paper, we present a novel Semantic Grouping Network, termed as SGN, that can directly disentangle sound representations and extract high-level semantic information for each source from input audio mixture. Specifically, SGN aggregates category-wise source features through learnable class tokens of sounds. Then, the aggregated semantic features can be used as the guidance to separate the corresponding audio sources from the mixture. We conducted extensive experiments on music-only and universal sound separation benchmarks: MUSIC, FUSS, MUSDB18, and VGG-Sound. The results demonstrate that our SGN significantly outperforms previous audio-only methods and audio-visual models without utilizing additional visual cues.

Figures (5)

• Figure 1: Illustration of our Semantic Grouping Network (SGN). The Category-aware Grouping module takes as input raw features of the input mixture spectrogram $\{\mathbf{f}_p\}_{p=1}^P$ and learnable class tokens $\{\mathbf{c}_i\}_{i=1}^C$ of for $C$ categories in the semantic space to generate disentangled class-aware representations $\{\mathbf{g}_i\}_{i=1}^N$ for $N$ sources. Note that $N$ source embeddings are chosen according to the ground-truth class. With the category-aware embeddings $\{\mathbf{g}_i\}_{i=1}^N$ and U-Net features of the mixture spectrogram, a light reconstruction network is used to reconstruct source spectrograms. Finally, inverse STFT is applied to recover the waveform of each source from spectrograms.
• Figure 2: Qualitative comparisons with audio-visual and audio-only baselines zhao2018theKong2021DecouplingMA. The proposed SGN achieves much better separation performance in terms of the quality of reconstructed source spectrograms. To better illustrate the superiority, some regions are highlighted by red boxes.
• Figure 3: Qualitative comparisons of representations learned by SoP, ResUNetDecouple+, and the proposed SGN. Note that each spot denotes the feature of one source sound, and each color refers to one category, such as "acoustic_guitar" in yellow and "erhu" in red.
• Figure 4: Quantitative results (Precision, Recall, and F1 score) of learned source class tokens.
• Figure 5: Visualization results of separated source spectrograms.