Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

DisContSE: Single-Step Diffusion Speech Enhancement Based on Joint Discrete and Continuous Embeddings

Yihui Fu, Tim Fingscheidt

TL;DR

DisContSE tackles the computational burden of diffusion-based speech enhancement by fusing discrete codec-token enhancement, continuous embedding refinement, and a semantic module within a single-step reverse diffusion framework. It introduces a quantization-error based mask initialization and a MaskGIT-style training regime to enable efficient inference, while freezing large pre-trained encoders. The approach yields strong improvements across intrusive and non-intrusive metrics and subjective MOS on URGENT 2024 data, with ablations confirming the value of each component. This work offers a practical pathway to high-fidelity, phoneme-conscious speech enhancement suitable for real-time or low-latency applications. The combination of discrete tokens, continuous embeddings, and semantic guidance represents a notable advance in diffusion-based SE, potentially influencing future codec- and embedding-enabled denoising approaches.

Abstract

Diffusion speech enhancement on discrete audio codec features gain immense attention due to their improved speech component reconstruction capability. However, they usually suffer from high inference computational complexity due to multiple reverse process iterations. Furthermore, they generally achieve promising results on non-intrusive metrics but show poor performance on intrusive metrics, as they may struggle in reconstructing the correct phones. In this paper, we propose DisContSE, an efficient diffusion-based speech enhancement model on joint discrete codec tokens and continuous embeddings. Our contributions are three-fold. First, we formulate both a discrete and a continuous enhancement module operating on discrete audio codec tokens and continuous embeddings, respectively, to achieve improved fidelity and intelligibility simultaneously. Second, a semantic enhancement module is further adopted to achieve optimal phonetic accuracy. Third, we achieve a single-step efficient reverse process in inference with a novel quantization error mask initialization strategy, which, according to our knowledge, is the first successful single-step diffusion speech enhancement based on an audio codec. Trained and evaluated on URGENT 2024 Speech Enhancement Challenge data splits, the proposed DisContSE excels top-reported time- and frequency-domain diffusion baseline methods in PESQ, POLQA, UTMOS, and in a subjective ITU-T P.808 listening test, clearly achieving an overall top rank.

DisContSE: Single-Step Diffusion Speech Enhancement Based on Joint Discrete and Continuous Embeddings

TL;DR

DisContSE tackles the computational burden of diffusion-based speech enhancement by fusing discrete codec-token enhancement, continuous embedding refinement, and a semantic module within a single-step reverse diffusion framework. It introduces a quantization-error based mask initialization and a MaskGIT-style training regime to enable efficient inference, while freezing large pre-trained encoders. The approach yields strong improvements across intrusive and non-intrusive metrics and subjective MOS on URGENT 2024 data, with ablations confirming the value of each component. This work offers a practical pathway to high-fidelity, phoneme-conscious speech enhancement suitable for real-time or low-latency applications. The combination of discrete tokens, continuous embeddings, and semantic guidance represents a notable advance in diffusion-based SE, potentially influencing future codec- and embedding-enabled denoising approaches.

Abstract

Diffusion speech enhancement on discrete audio codec features gain immense attention due to their improved speech component reconstruction capability. However, they usually suffer from high inference computational complexity due to multiple reverse process iterations. Furthermore, they generally achieve promising results on non-intrusive metrics but show poor performance on intrusive metrics, as they may struggle in reconstructing the correct phones. In this paper, we propose DisContSE, an efficient diffusion-based speech enhancement model on joint discrete codec tokens and continuous embeddings. Our contributions are three-fold. First, we formulate both a discrete and a continuous enhancement module operating on discrete audio codec tokens and continuous embeddings, respectively, to achieve improved fidelity and intelligibility simultaneously. Second, a semantic enhancement module is further adopted to achieve optimal phonetic accuracy. Third, we achieve a single-step efficient reverse process in inference with a novel quantization error mask initialization strategy, which, according to our knowledge, is the first successful single-step diffusion speech enhancement based on an audio codec. Trained and evaluated on URGENT 2024 Speech Enhancement Challenge data splits, the proposed DisContSE excels top-reported time- and frequency-domain diffusion baseline methods in PESQ, POLQA, UTMOS, and in a subjective ITU-T P.808 listening test, clearly achieving an overall top rank.
Paper Structure (12 sections, 3 equations, 3 figures, 2 tables)

This paper contains 12 sections, 3 equations, 3 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Methods
  3. Experimental Setup
  4. Networks
  5. Data and Metrics
  6. Model Training
  7. Results and Discussion
  8. Main Results
  9. Ablation Study
  10. Conclusions
  11. Acknowledgment
  12. Compliance with Ethical Standards

Figures (3)

  • Figure 1: (a) Architecture of the proposed DisContSE and its training strategy. (b) Block diagram of the continuous enhancement module and its MAE loss. (c) Block diagram of the semantic enhancement module and its MAE loss.
  • Figure 2: Inference of the proposed DisContSE.
  • Figure 3: DisContSE performance on $T$ in the single-step reverse process (inference) on $\mathcal{D}^{\textrm{val}}$.