Diffusion-based Frameworks for Unsupervised Speech Enhancement

TL;DR

The work tackles unsupervised single-channel speech enhancement by leveraging diffusion models with explicit latent-variable noise modeling. It introduces DiffUSEEN, where speech and noise are jointly sampled under an NMF-based noise prior, and ParaDiffUSE, which learns a shared diffusion prior for both speech and noise with two variants: implicit (ParaDiffUSE-IN) and explicit noise sampling (ParaDiffUSE-EN). Results show that explicit noise modeling improves data-consistency and SE quality, with ParaDiffUSE-EN achieving top performance among unsupervised methods under matched conditions and DiffUSEEN offering greater robustness under mismatches. The approaches reduce reliance on paired clean–noisy data and offer practical pathways toward robust, unsupervised SE, with publicly available code planned.

Abstract

This paper addresses $\textit{unsupervised}$ diffusion-based single-channel speech enhancement (SE). Prior work in this direction combines a score-based diffusion model trained on clean speech with a Gaussian noise model whose covariance is structured by non-negative matrix factorization (NMF). This combination is used within an iterative expectation-maximization (EM) scheme, in which a diffusion-based posterior-sampling E-step estimates the clean speech. We first revisit this framework and propose to explicitly model both speech and acoustic noise as latent variables, jointly sampling them in the E-step instead of sampling speech alone as in previous approaches. We then introduce a new unsupervised SE framework that replaces the NMF noise prior with a diffusion-based noise model, learned jointly with the speech prior in a single conditional score model. Within this framework, we derive two variants: one that implicitly accounts for noise and one that explicitly treats noise as a latent variable. Experiments on WSJ0-QUT and VoiceBank-DEMAND show that explicit noise modeling systematically improves SE performance for both NMF-based and diffusion-based noise priors. Under matched conditions, the diffusion-based noise model attains the best overall quality and intelligibility among unsupervised methods, while under mismatched conditions the proposed NMF-based explicit-noise framework is more robust and suffers less degradation than several supervised baselines. Our code will be publicly available on this $\href{https://github.com/jeaneudesAyilo/enudiffuse}{URL}$.

Figures (4)

• Figure 1: Schematic diagram of the DiffUSEEN algorithm.
• Figure 2: Schematic diagram of the ParaDiffUSE algorithm.
• Figure 3: Violin plots showing the SI-SDR distributions for the matched and mismatched conditions on the VB-DMD test set, with dashed and dotted lines indicating the median and quartiles, respectively. For readability, we omit the RemixIT and DEPSE-IL violins: the former has an SI-SDR range far from the other methods, and the latter exhibits a trend very similar to UDiffSE+.
• Figure 4: Effect of noise-estimation setting (Estimated / Oracle / Pure Oracle) for UDiffSE+, DiffUSEEN, and ParaDiffUSE-EN on WSJ0–QUT.