SSNAPS: Audio-Visual Separation of Speech and Background Noise with Diffusion Inverse Sampling

TL;DR

SSNAPS tackles unsupervised audio-visual single-microphone speech separation by modeling clean speech and ambient noise as diffusion priors $p({oldsymbol{x}}_0)$ and $p({\boldsymbol{n}}_0)$ and sampling from the posterior $p({\boldsymbol{c}}, {\boldsymbol{n}}|{\boldsymbol{y}})$ via diffusion inverse sampling. It extends the Decoupled Annealing Posterior Sampling (DAPS) framework to handle multiple sources drawn from two distributions, conditioning speech scores on lip-video features ${\boldsymbol{V}}$ through FiLM, and jointly reconstructing noise via a spectral-domain likelihood ${\boldsymbol{\\mathcal{L}}_{rec}}$. The approach achieves best WER across 1–3 speaker mixtures with noise on VoxCeleb2+DNS and VoxCeleb2+DCASE, while enabling an off-screen extension and producing high-fidelity ambient noise suitable for acoustic scene detection. This unsupervised, diffusion-based framework reduces reliance on degradation-specific labeled data and provides a flexible, scalable solution for robust speech separation and downstream analysis in real-world environments.

Abstract

This paper addresses the challenge of audio-visual single-microphone speech separation and enhancement in the presence of real-world environmental noise. Our approach is based on generative inverse sampling, where we model clean speech and ambient noise with dedicated diffusion priors and jointly leverage them to recover all underlying sources. To achieve this, we reformulate a recent inverse sampler to match our setting. We evaluate on mixtures of 1, 2, and 3 speakers with noise and show that, despite being entirely unsupervised, our method consistently outperforms leading supervised baselines in \ac{WER} across all conditions. We further extend our framework to handle off-screen speaker separation. Moreover, the high fidelity of the separated noise component makes it suitable for downstream acoustic scene detection. Demo page: https://ssnapsicml.github.io/ssnapsicml2026/