Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Diffusion-based Generative Modeling with Discriminative Guidance for Streamable Speech Enhancement

Chenda Li, Samuele Cornell, Shinji Watanabe, Yanmin Qian

TL;DR

This work tackles real-time speech enhancement with diffusion-based generative models by introducing discriminative guidance to drastically reduce reverse-diffusion steps and proposing a streamable, time-domain, chunk-level diffusion framework. By warm-starting the diffusion with a discriminative score for the first $N_ phi$ steps and maintaining history across chunks with SkiM, the approach achieves a favorable trade-off between generative quality and computational efficiency. Empirical results on WSJ0-CHiME3 and CHiME4 show that online performance can approach offline baselines on in-domain data while improving generalization to real-world data, with substantial reductions in MACs as $N_ phi$ increases. The proposed framework yields a practical path toward low-latency, robust streaming SE, with clear directions for latency reduction, broader applicability, and refined balance strategies between generative and discriminative components.

Abstract

Diffusion-based generative models (DGMs) have recently attracted attention in speech enhancement research (SE) as previous works showed a remarkable generalization capability. However, DGMs are also computationally intensive, as they usually require many iterations in the reverse diffusion process (RDP), making them impractical for streaming SE systems. In this paper, we propose to use discriminative scores from discriminative models in the first steps of the RDP. These discriminative scores require only one forward pass with the discriminative model for multiple RDP steps, thus greatly reducing computations. This approach also allows for performance improvements. We show that we can trade off between generative and discriminative capabilities as the number of steps with the discriminative score increases. Furthermore, we propose a novel streamable time-domain generative model with an algorithmic latency of 50 ms, which has no significant performance degradation compared to offline models.

Diffusion-based Generative Modeling with Discriminative Guidance for Streamable Speech Enhancement

TL;DR

This work tackles real-time speech enhancement with diffusion-based generative models by introducing discriminative guidance to drastically reduce reverse-diffusion steps and proposing a streamable, time-domain, chunk-level diffusion framework. By warm-starting the diffusion with a discriminative score for the first steps and maintaining history across chunks with SkiM, the approach achieves a favorable trade-off between generative quality and computational efficiency. Empirical results on WSJ0-CHiME3 and CHiME4 show that online performance can approach offline baselines on in-domain data while improving generalization to real-world data, with substantial reductions in MACs as increases. The proposed framework yields a practical path toward low-latency, robust streaming SE, with clear directions for latency reduction, broader applicability, and refined balance strategies between generative and discriminative components.

Abstract

Diffusion-based generative models (DGMs) have recently attracted attention in speech enhancement research (SE) as previous works showed a remarkable generalization capability. However, DGMs are also computationally intensive, as they usually require many iterations in the reverse diffusion process (RDP), making them impractical for streaming SE systems. In this paper, we propose to use discriminative scores from discriminative models in the first steps of the RDP. These discriminative scores require only one forward pass with the discriminative model for multiple RDP steps, thus greatly reducing computations. This approach also allows for performance improvements. We show that we can trade off between generative and discriminative capabilities as the number of steps with the discriminative score increases. Furthermore, we propose a novel streamable time-domain generative model with an algorithmic latency of 50 ms, which has no significant performance degradation compared to offline models.
Paper Structure (14 sections, 11 equations, 1 figure, 1 table)

This paper contains 14 sections, 11 equations, 1 figure, 1 table.

Table of Contents

  1. Introduction
  2. Methods
  3. Score-based generative speech enhancement
  4. Score guided by discriminative models
  5. Streamable chunk-Level diffusion-based SE
  6. Experimental Setups
  7. Dataset
  8. Evaluation metrics
  9. Model configurations
  10. Results and Discussions
  11. Results of online SGMSE and algorithm latency
  12. Results with discriminative guidance
  13. Conclusion and Future Works
  14. Acknowledgment

Figures (1)

  • Figure 1: (A)-(D): Effect of proposed discriminative guidance on WSJ0-CHiME3 and CHiME4(real) test set. The total number of reverse steps is $N=30$, $\mathbf{s}_{\theta}$ is replaced by $\mathbf{s}_{D}$ in the first $N_{\phi}$ steps of $N$; (E): GMACs/s as $N_{\phi}$ increases; the 14.4 GMACs/s for $N_{\phi}=30$ corresponds to using only SkiM-D (Sys.3).