Complex-Cycle-Consistent Diffusion Model for Monaural Speech Enhancement

TL;DR

The paper tackles monaural speech enhancement in real-world noisy environments by introducing a diffusion-based framework, SEDM, that separately estimates speech magnitude and phase and uses real-world noise in the forward diffusion. A noise-aware reverse process jointly recovers clean speech and noise spectra, and a Complex-Cycle-Consistent (CCC) learning module enforces algebraic relationships between magnitude and phase via LSTM-based mappings and cycle-consistent losses. The authors provide extensive experiments across IEEE, TIMIT, VCTK, and DNS datasets, showing SEDM-L achieves state-of-the-art or competitive performance across many objective metrics and demonstrating the value of modeling real-world noise and magnitude–phase coupling. Overall, the work highlights the benefits of phase-aware diffusion and magnitude–phase consistency for robust, high-quality monaural speech enhancement with practical implications for noisy audio processing systems.

Abstract

In this paper, we present a novel diffusion model-based monaural speech enhancement method. Our approach incorporates the separate estimation of speech spectra's magnitude and phase in two diffusion networks. Throughout the diffusion process, noise clips from real-world noise interferences are added gradually to the clean speech spectra and a noise-aware reverse process is proposed to learn how to generate both clean speech spectra and noise spectra. Furthermore, to fully leverage the intrinsic relationship between magnitude and phase, we introduce a complex-cycle-consistent (CCC) mechanism that uses the estimated magnitude to map the phase, and vice versa. We implement this algorithm within a phase-aware speech enhancement diffusion model (SEDM). We conduct extensive experiments on public datasets to demonstrate the effectiveness of our method, highlighting the significant benefits of exploiting the intrinsic relationship between phase and magnitude information to enhance speech. The comparison to conventional diffusion models demonstrates the superiority of SEDM.

Figures (7)

• Figure 1: The overall architecture of SEDM consisted of a diffusion process (left) and a noise-aware reverse process (right).
• Figure 2: The proposed diffusion block and reverse block.
• Figure 3: The proposed complex-cycle-consistent learning for speech (CCC) enhancement.
• Figure 4: The pipeline of (diffusion model: $\text{✓}$, phase-aware: $\text{✗}$, and CCC: $\text{✓}$). The clean speech spectra and the corresponding reconstruction are only converted into magnitude and phase components before CCC module.
• Figure 5: IEEE dataset visualization with diffusion t-SNE for different numbers of embeddings N.