Combining Deterministic Enhanced Conditions with Dual-Streaming Encoding for Diffusion-Based Speech Enhancement

TL;DR

This work tackles the challenge of providing reliable priors for diffusion-based speech enhancement by leveraging deterministic, enhanced features as conditioning signals. It introduces DERDM-SE, a framework that unifies deterministic and diffusion models through a dual-stream encoder and two key components: COFFEE, a granularity-progressive deterministic model, and D2U-SE, a Deterministic-Diffusion Unified SE system with a dual-decoder design. The paper extends to D3U-SE for simultaneous use of deterministic-only and deterministic-noisy conditions, and demonstrates through CHiME4 experiments that these approaches can achieve improved SE scores and more stable diffusion performance compared to existing baselines. The findings highlight that the choice of deterministic model and conditioning strategy greatly influences diffusion performance on real versus simulated data, with COFFEE providing robust gains in MOS metrics and D2U-SE offering stable objective improvements across several configurations.

Abstract

Diffusion-based speech enhancement (SE) models need to incorporate correct prior knowledge as reliable conditions to generate accurate predictions. However, providing reliable conditions using noisy features is challenging. One solution is to use features enhanced by deterministic methods as conditions. However, the information distortion and loss caused by deterministic methods might affect the diffusion process. In this paper, we first investigate the effects of using different deterministic SE models as conditions for diffusion. We validate two conditions depending on whether the noisy feature was used as part of the condition: one using only the deterministic feature (deterministic-only), and the other using both deterministic and noisy features (deterministic-noisy). Preliminary investigation found that using deterministic enhanced conditions improves hearing experiences on real data, while the choice between using deterministic-only or deterministic-noisy conditions depends on the deterministic models. Based on these findings, we propose a dual-streaming encoding Repair-Diffusion Model for SE (DERDM-SE) to more effectively utilize both conditions. Moreover, we found that fine-grained deterministic models have greater potential in objective evaluation metrics, while UNet-based deterministic models provide more stable diffusion performance. Therefore, in the DERDM-SE, we propose a deterministic model that combines coarse- and fine-grained processing. Experimental results on CHiME4 show that the proposed models effectively leverage deterministic models to achieve better SE evaluation scores, along with more stable performance compared to other diffusion-based SE models.

Figures (6)

• Figure 1: High-level conceptual flowchart of the D2U-SE framework. Each stage corresponds to a core modeling motivation.
• Figure 2: Flowchart of the proposed method. It includes the proposed COFFEE deterministic model and the proposed D2U-SE. Within COFFEE, we introduce Time-Embedding (T-E) bin-level, Time-Reconstructed Feature (T-RF) bin-level, and Time-Frequency (T-F) bin-level processing. Within D2U-SE, the skip connections between the encoder and decoder are omitted in the figure for clarity. Lowercase letters denote the time-domain representations of their corresponding uppercase symbols.
• Figure 3: Flowchart of the proposed D3U-SE. The skip connections between the encoder and decoder are omitted in the figure for clarity. Lowercase letters denote the time-domain representations of their corresponding uppercase symbols.
• Figure 4: dMOS scores under different noise scenarios using various SE systems on the real evaluation set of the CHiME4 dataset. In each group, models below the red dashed line exhibit a statistically significant performance improvement (significance computed as described in Section \ref{['sec:evalution_desc']}). Bars marked with denote the proposed model.
• Figure 5: uMOS scores under different noise scenarios using various SE systems on the real evaluation set of the CHiME4 dataset. In each group, models below the red dashed line exhibit a statistically significant performance improvement (significance computed as described in Section \ref{['sec:evalution_desc']}). Bars marked with denote the proposed model.