Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Absorbing Discrete Diffusion for Speech Enhancement

Philippe Gonzalez

TL;DR

To efficiently model the hierarchical structure of residual vector quantization codes, this work proposes RQDiT, which combines techniques from RQ-Transformer and diffusion Transformers for non-autoregressive modeling.

Abstract

Inspired by recent developments in neural speech coding and diffusion-based language modeling, we tackle speech enhancement by modeling the conditional distribution of clean speech codes given noisy speech codes using absorbing discrete diffusion. The proposed approach, which we call ADDSE, leverages both the expressive latent space of neural audio codecs and the non-autoregressive sampling procedure of diffusion models. To efficiently model the hierarchical structure of residual vector quantization codes, we propose RQDiT, which combines techniques from RQ-Transformer and diffusion Transformers for non-autoregressive modeling. Results show competitive performance in terms of non-intrusive objective metrics on two datasets, especially at low signal-to-noise ratios and with few sampling steps. Code and audio examples are available online.

Absorbing Discrete Diffusion for Speech Enhancement

TL;DR

To efficiently model the hierarchical structure of residual vector quantization codes, this work proposes RQDiT, which combines techniques from RQ-Transformer and diffusion Transformers for non-autoregressive modeling.

Abstract

Inspired by recent developments in neural speech coding and diffusion-based language modeling, we tackle speech enhancement by modeling the conditional distribution of clean speech codes given noisy speech codes using absorbing discrete diffusion. The proposed approach, which we call ADDSE, leverages both the expressive latent space of neural audio codecs and the non-autoregressive sampling procedure of diffusion models. To efficiently model the hierarchical structure of residual vector quantization codes, we propose RQDiT, which combines techniques from RQ-Transformer and diffusion Transformers for non-autoregressive modeling. Results show competitive performance in terms of non-intrusive objective metrics on two datasets, especially at low signal-to-noise ratios and with few sampling steps. Code and audio examples are available online.
Paper Structure (13 sections, 9 equations, 4 figures, 1 table)

This paper contains 13 sections, 9 equations, 4 figures, 1 table.

Table of Contents

  1. Introduction
  2. Background
  3. Neural audio codecs and residual vector quantization
  4. Absorbing discrete diffusion
  5. Absorbing discrete diffusion for speech enhancement
  6. Noisy-speech-conditioned absorbing discrete diffusion
  7. Hierarchical non-autoregressive sequence modeling
  8. Experimental setup
  9. Datasets
  10. Neural audio codec
  11. Compared systems
  12. Results
  13. Conclusion

Figures (4)

  • Figure 1: Proposed ADDSE framework.
  • Figure 2: RQDiT architecture. Two DiTs are applied along the NAC frame and depth dimensions, respectively.
  • Figure 3: Top and middle rows: Non-intrusive metrics as a function of $N_\text{steps}$. Bottom left: Number of function evaluations as a function of $N_\text{steps}$. Bottom right: DCE on test data.
  • Figure 4: Non-intrusive metrics as a function of input SNR.