Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Diffusion-Based Speech Enhancement with Joint Generative and Predictive Decoders

Hao Shi, Kazuki Shimada, Masato Hirano, Takashi Shibuya, Yuichiro Koyama, Zhi Zhong, Shusuke Takahashi, Tatsuya Kawahara, Yuki Mitsufuji

TL;DR

This work addresses the slow decoding of diffusion-based speech enhancement by proposing a unified framework that jointly leverages generative and predictive decoders. The authors introduce two architectures, GP-SGMSE+ and GP-Unified, with a shared encoder and dual decoders, and implement fusion of predictive and generative features at the initial and final diffusion steps to speed up convergence and exploit complementary distortions. On Voice-Bank-DEMAND, the unified approach achieves faster diffusion and higher PESQ scores than prior score-based diffusion SE methods, while reducing the number of diffusion steps via informative initialization and late-stage fusion. The results highlight the practical impact of integrating predictive information into diffusion-based SE, yielding both efficiency gains and improved speech quality.

Abstract

Diffusion-based generative speech enhancement (SE) has recently received attention, but reverse diffusion remains time-consuming. One solution is to initialize the reverse diffusion process with enhanced features estimated by a predictive SE system. However, the pipeline structure currently does not consider for a combined use of generative and predictive decoders. The predictive decoder allows us to use the further complementarity between predictive and diffusion-based generative SE. In this paper, we propose a unified system that use jointly generative and predictive decoders across two levels. The encoder encodes both generative and predictive information at the shared encoding level. At the decoded feature level, we fuse the two decoded features by generative and predictive decoders. Specifically, the two SE modules are fused in the initial and final diffusion steps: the initial fusion initializes the diffusion process with the predictive SE to improve convergence, and the final fusion combines the two complementary SE outputs to enhance SE performance. Experiments conducted on the Voice-Bank dataset demonstrate that incorporating predictive information leads to faster decoding and higher PESQ scores compared with other score-based diffusion SE (StoRM and SGMSE+).

Diffusion-Based Speech Enhancement with Joint Generative and Predictive Decoders

TL;DR

This work addresses the slow decoding of diffusion-based speech enhancement by proposing a unified framework that jointly leverages generative and predictive decoders. The authors introduce two architectures, GP-SGMSE+ and GP-Unified, with a shared encoder and dual decoders, and implement fusion of predictive and generative features at the initial and final diffusion steps to speed up convergence and exploit complementary distortions. On Voice-Bank-DEMAND, the unified approach achieves faster diffusion and higher PESQ scores than prior score-based diffusion SE methods, while reducing the number of diffusion steps via informative initialization and late-stage fusion. The results highlight the practical impact of integrating predictive information into diffusion-based SE, yielding both efficiency gains and improved speech quality.

Abstract

Diffusion-based generative speech enhancement (SE) has recently received attention, but reverse diffusion remains time-consuming. One solution is to initialize the reverse diffusion process with enhanced features estimated by a predictive SE system. However, the pipeline structure currently does not consider for a combined use of generative and predictive decoders. The predictive decoder allows us to use the further complementarity between predictive and diffusion-based generative SE. In this paper, we propose a unified system that use jointly generative and predictive decoders across two levels. The encoder encodes both generative and predictive information at the shared encoding level. At the decoded feature level, we fuse the two decoded features by generative and predictive decoders. Specifically, the two SE modules are fused in the initial and final diffusion steps: the initial fusion initializes the diffusion process with the predictive SE to improve convergence, and the final fusion combines the two complementary SE outputs to enhance SE performance. Experiments conducted on the Voice-Bank dataset demonstrate that incorporating predictive information leads to faster decoding and higher PESQ scores compared with other score-based diffusion SE (StoRM and SGMSE+).
Paper Structure (14 sections, 12 equations, 7 figures, 3 tables)

This paper contains 14 sections, 12 equations, 7 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Score-based Diffusion Model
  3. Stochastic process
  4. Training objective
  5. Inference
  6. Proposed Method
  7. GP-SGMSE+
  8. GP-Unified
  9. Experimental Evaluations
  10. Effect of incorporating predictive loss function
  11. Effect of the first and final fusion
  12. Effect of fusion hyperparameter $\beta$
  13. Comparasion with other methods
  14. Conclusions and Future Works

Figures (7)

  • Figure 1: The score model (used in PC samplers) structure: (a) the baseline SGMSE+; (b) the proposed Generative and Predictive based SGMSE+ (GP-SGMSE+); (c) the proposed Unified Generative and Predictive model (GP-Unified). Note that the skip connection exists between the encoder and decoders (generative and predictive).
  • Figure 2: A flowchart of the proposed method. Generative and predictive SE systems are fused in the first and final diffusion steps.
  • Figure 3: PESQ performance with different diffusion steps
  • Figure 4: SI-SDR performance with different diffusion steps
  • Figure 5: SI-SIR performance with different diffusion steps
  • ...and 2 more figures