Unsupervised Single-Channel Audio Separation with Diffusion Source Priors
Runwu Shi, Chang Li, Jiang Wang, Rui Zhang, Nabeela Khan, Benjamin Yen, Takeshi Ashizawa, Kazuhiro Nakadai
TL;DR
This work tackles unsupervised single-channel audio separation by casting it as a diffusion inverse problem that leverages diffusion priors trained on individual sources. It introduces a hybrid guidance strength schedule to mitigate gradient conflicts between prior and likelihood, and a noise-augmented mixture initialization to ground the reverse process in the observed mixture. A novel TF-domain, triple-path self-attention diffusion backbone operates directly in waveform space to learn rich audio priors, enabling strong performance across speech and sound-event separation tasks. Across speech-sound, sound-event, and pure speech separation, the approach yields significant improvements over prior unsupervised methods and remains competitive with fully supervised baselines, highlighting data-efficient generalization capabilities. Collectively, these contributions advance unsupervised diffusion-based audio separation and demonstrate robust applicability to real-world multi-source scenarios.
Abstract
Single-channel audio separation aims to separate individual sources from a single-channel mixture. Most existing methods rely on supervised learning with synthetically generated paired data. However, obtaining high-quality paired data in real-world scenarios is often difficult. This data scarcity can degrade model performance under unseen conditions and limit generalization ability. To this end, in this work, we approach this problem from an unsupervised perspective, framing it as a probabilistic inverse problem. Our method requires only diffusion priors trained on individual sources. Separation is then achieved by iteratively guiding an initial state toward the solution through reconstruction guidance. Importantly, we introduce an advanced inverse problem solver specifically designed for separation, which mitigates gradient conflicts caused by interference between the diffusion prior and reconstruction guidance during inverse denoising. This design ensures high-quality and balanced separation performance across individual sources. Additionally, we find that initializing the denoising process with an augmented mixture instead of pure Gaussian noise provides an informative starting point that significantly improves the final performance. To further enhance audio prior modeling, we design a novel time-frequency attention-based network architecture that demonstrates strong audio modeling capability. Collectively, these improvements lead to significant performance gains, as validated across speech-sound event, sound event, and speech separation tasks.