Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Speaking in Wavelet Domain: A Simple and Efficient Approach to Speed up Speech Diffusion Model

Xiangyu Zhang, Daijiao Liu, Hexin Liu, Qiquan Zhang, Hanyu Meng, Leibny Paola Garcia, Eng Siong Chng, Lina Yao

TL;DR

The paper tackles the slow training and inference costs of speech diffusion models by shifting the generative target to the wavelet domain. By applying a Discrete Wavelet Transform (CDF 5/3) to obtain $c_A$ and $c_D$ and training on the concatenated $y=[c_A;c_D]$, the approach halves data length and memory while maintaining or improving speech quality. It demonstrates effectiveness on both synthesis and enhancement tasks (DiffWave and CDiffuSE) and introduces a Low-frequency Enhancer and a Multi-Level Wavelet Accelerator to push speedups beyond 5× with modest quality changes. This wavelet-centric, front-end strategy offers a practical, model-agnostic path to faster, scalable diffusion-based speech systems and highlights task-dependent wavelet choices for optimal results.

Abstract

Recently, Denoising Diffusion Probabilistic Models (DDPMs) have attained leading performances across a diverse range of generative tasks. However, in the field of speech synthesis, although DDPMs exhibit impressive performance, their long training duration and substantial inference costs hinder practical deployment. Existing approaches primarily focus on enhancing inference speed, while approaches to accelerate training a key factor in the costs associated with adding or customizing voices often necessitate complex modifications to the model, compromising their universal applicability. To address the aforementioned challenges, we propose an inquiry: is it possible to enhance the training/inference speed and performance of DDPMs by modifying the speech signal itself? In this paper, we double the training and inference speed of Speech DDPMs by simply redirecting the generative target to the wavelet domain. This method not only achieves comparable or superior performance to the original model in speech synthesis tasks but also demonstrates its versatility. By investigating and utilizing different wavelet bases, our approach proves effective not just in speech synthesis, but also in speech enhancement.

Speaking in Wavelet Domain: A Simple and Efficient Approach to Speed up Speech Diffusion Model

TL;DR

The paper tackles the slow training and inference costs of speech diffusion models by shifting the generative target to the wavelet domain. By applying a Discrete Wavelet Transform (CDF 5/3) to obtain and and training on the concatenated , the approach halves data length and memory while maintaining or improving speech quality. It demonstrates effectiveness on both synthesis and enhancement tasks (DiffWave and CDiffuSE) and introduces a Low-frequency Enhancer and a Multi-Level Wavelet Accelerator to push speedups beyond 5× with modest quality changes. This wavelet-centric, front-end strategy offers a practical, model-agnostic path to faster, scalable diffusion-based speech systems and highlights task-dependent wavelet choices for optimal results.

Abstract

Recently, Denoising Diffusion Probabilistic Models (DDPMs) have attained leading performances across a diverse range of generative tasks. However, in the field of speech synthesis, although DDPMs exhibit impressive performance, their long training duration and substantial inference costs hinder practical deployment. Existing approaches primarily focus on enhancing inference speed, while approaches to accelerate training a key factor in the costs associated with adding or customizing voices often necessitate complex modifications to the model, compromising their universal applicability. To address the aforementioned challenges, we propose an inquiry: is it possible to enhance the training/inference speed and performance of DDPMs by modifying the speech signal itself? In this paper, we double the training and inference speed of Speech DDPMs by simply redirecting the generative target to the wavelet domain. This method not only achieves comparable or superior performance to the original model in speech synthesis tasks but also demonstrates its versatility. By investigating and utilizing different wavelet bases, our approach proves effective not just in speech synthesis, but also in speech enhancement.
Paper Structure (27 sections, 17 equations, 4 figures, 4 tables, 2 algorithms)

This paper contains 27 sections, 17 equations, 4 figures, 4 tables, 2 algorithms.

Table of Contents

  1. Introduction
  2. Related Work
  3. Methodology
  4. Wavelet Transform and Compression
  5. Wavelet-based Speech Diffusion Scheme
  6. Speech Synthesis
  7. Speech Enhancement
  8. Experiments
  9. Dataset
  10. Model Architecture and Training
  11. Main Result
  12. Further Study
  13. Low-frequency Speech Enhancer
  14. Multi-Level Wavelet Accelerator
  15. Ablation Study and Analysis
  16. ...and 12 more sections

Figures (4)

  • Figure 1: Wavelet of Cohen-Daubechies-Feauveau 5-tap/3-tap. (a) Scaling and wavelet functions, (b) decomposition and reconstruction filters.
  • Figure 2: Overview of the Speech Wavelet Diffusion Model pipeline: First, the speech signal is decomposed into Approximation coefficients Matrix(cA) and Detail coefficients matrix(cD), the Diffusion model subsequently generates cA and cD and restores the speech signal from these matrices.
  • Figure 3: Overview of (a) Block of Multi-Level Discrete Wavelet Transform, (b) Multi-Level Low-Frequency Voice Enhancement Module, (c) Block of Multi-Level Inverse Discrete Wavelet Transform.
  • Figure 4: Overview of Frequency Bottleneck Block