Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

On the Design of Diffusion-based Neural Speech Codecs

Pietro Foti, Andreas Brendel

TL;DR

This work systematically investigates diffusion-based neural speech codecs (NSCs) by introducing a conditioning/output-domain design space and evaluating six configurations (waveform, mel, and latent domains with various conditioning sources). Using DiffWave and GradTTS backbones, the authors compare diffusion-based codecs to GAN- and diffusion-based baselines, report objective metrics (ViSQOL, SCOREQ) and subjective listening tests on LibriTTS and VCTK data, and reveal that mel2mel (mel diffusion with mel-to-mel output) performs best among DM designs at 3 kbps, though a retrained encoder-quantizer baseline (EC) typically yields the strongest overall performance. The study highlights design tradeoffs, such as the computational complexity of waveform diffusion versus the interpretability of mel/latent diffusion, and provides a concrete methodology for evaluating and tuning diffusion-based NSCs. Overall, while diffusion-based NSCs approach strong baselines in quality, state-of-the-art performance is still dominated by optimized quantized encoder baselines, motivating further refinements in conditioning, quantization, and vocoding components.

Abstract

Recently, neural speech codecs (NSCs) trained as generative models have shown superior performance compared to conventional codecs at low bitrates. Although most state-of-the-art NSCs are trained as Generative Adversarial Networks (GANs), Diffusion Models (DMs), a recent class of generative models, represent a promising alternative due to their superior performance in image generation relative to GANs. Consequently, DMs have been successfully applied for audio and speech coding among various other audio generation applications. However, the design of diffusion-based NSCs has not yet been explored in a systematic way. We address this by providing a comprehensive analysis of diffusion-based NSCs divided into three contributions. First, we propose a categorization based on the conditioning and output domains of the DM. This simple conceptual framework allows us to define a design space for diffusion-based NSCs and to assign a category to existing approaches in the literature. Second, we systematically investigate unexplored designs by creating and evaluating new diffusion-based NSCs within the conceptual framework. Finally, we compare the proposed models to existing GAN and DM baselines through objective metrics and subjective listening tests.

On the Design of Diffusion-based Neural Speech Codecs

TL;DR

This work systematically investigates diffusion-based neural speech codecs (NSCs) by introducing a conditioning/output-domain design space and evaluating six configurations (waveform, mel, and latent domains with various conditioning sources). Using DiffWave and GradTTS backbones, the authors compare diffusion-based codecs to GAN- and diffusion-based baselines, report objective metrics (ViSQOL, SCOREQ) and subjective listening tests on LibriTTS and VCTK data, and reveal that mel2mel (mel diffusion with mel-to-mel output) performs best among DM designs at 3 kbps, though a retrained encoder-quantizer baseline (EC) typically yields the strongest overall performance. The study highlights design tradeoffs, such as the computational complexity of waveform diffusion versus the interpretability of mel/latent diffusion, and provides a concrete methodology for evaluating and tuning diffusion-based NSCs. Overall, while diffusion-based NSCs approach strong baselines in quality, state-of-the-art performance is still dominated by optimized quantized encoder baselines, motivating further refinements in conditioning, quantization, and vocoding components.

Abstract

Recently, neural speech codecs (NSCs) trained as generative models have shown superior performance compared to conventional codecs at low bitrates. Although most state-of-the-art NSCs are trained as Generative Adversarial Networks (GANs), Diffusion Models (DMs), a recent class of generative models, represent a promising alternative due to their superior performance in image generation relative to GANs. Consequently, DMs have been successfully applied for audio and speech coding among various other audio generation applications. However, the design of diffusion-based NSCs has not yet been explored in a systematic way. We address this by providing a comprehensive analysis of diffusion-based NSCs divided into three contributions. First, we propose a categorization based on the conditioning and output domains of the DM. This simple conceptual framework allows us to define a design space for diffusion-based NSCs and to assign a category to existing approaches in the literature. Second, we systematically investigate unexplored designs by creating and evaluating new diffusion-based NSCs within the conceptual framework. Finally, we compare the proposed models to existing GAN and DM baselines through objective metrics and subjective listening tests.

Paper Structure

This paper contains 15 sections, 2 equations, 4 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Diffusion Model-based Speech Codecs
  3. Experimental Setup
  4. Model Design
  5. Training and Evaluation
  6. Preliminary Experiments
  7. Mel-spectrogram diffusion
  8. Latent diffusion
  9. Experiments
  10. Exp. 1 - Evaluation of proposed designs
  11. Exp. 2 - Best proposed design vs baselines
  12. Exp. 3 - Best performers at different bitrates
  13. Listening Tests
  14. Experimental Results
  15. Conclusion

Figures (4)

  • Figure 1: Sampling scheme of the proposed dm-based nscs. The encoder can be fixed (mel) or learned (lat).
  • Figure 2: Objective evaluation of Exp. 1 (a), 2 (b) and 3 (c). (b) shows retrained models on the left of the dashed line, pretrained ones on the right, with gan and dm models colored in red and blue respectively.
  • Figure 3: P.808 DCR test results (including 15 listeners) comparing the proposed dm-based nscs at 3 kbps.
  • Figure 4: P.808 DCR test results (including 19 listeners) comparing mel2mel to gan and dm baselines.