Audio Generation Through Score-Based Generative Modeling: Design Principles and Implementation

TL;DR

This paper benchmarks diffusion-based audio generation through the score-based diffusion and EDM frameworks, presenting principled design guidelines for training, sampling, and architecture to improve audio quality and conditioning flexibility. It connects DSM training, SDE/PF-ODE formulations, and modular conditioning to practical audio applications, from text-to-audio and TTS to restoration and source separation. A comprehensive open-source codebase accompanies the survey, enabling reproducible experimentation across tasks and architectures. Case studies on unconditional generation, speech enhancement, and text-to-speech demonstrate the framework's versatility, performance trade-offs, and scalability, highlighting the framework's potential to accelerate real-world audio synthesis research and deployment.

Abstract

Diffusion models have emerged as powerful deep generative techniques, producing high-quality and diverse samples in applications in various domains including audio. While existing reviews provide overviews, there remains limited in-depth discussion of these specific design choices. The audio diffusion model literature also lacks principled guidance for the implementation of these design choices and their comparisons for different applications. This survey provides a comprehensive review of diffusion model design with an emphasis on design principles for quality improvement and conditioning for audio applications. We adopt the score modeling perspective as a unifying framework that accommodates various interpretations, including recent approaches like flow matching. We systematically examine the training and sampling procedures of diffusion models, and audio applications through different conditioning mechanisms. To provide an integrated, unified codebase and to promote reproducible research and rapid prototyping, we introduce an open-source codebase (https://github.com/gzhu06/AudioDiffuser) that implements our reviewed framework for various audio applications. We demonstrate its capabilities through three case studies: audio generation, speech enhancement, and text-to-speech synthesis, with benchmark evaluations on standard datasets.

Figures (7)

• Figure 1: Design space of diffusion-based generative models for audio applications. The diagram illustrates the key modular components for diffusion models, organized into three categories: (1) training components, including training objectives (noise/velocity/data prediction, flow matching) and noise schedules (uniform, cosine-uniform, sigmoid-uniform, etc.); (2) sampling components, including deterministic samplers (Euler, Heun, DEIS, DPM-Solver, RK-Solver), sampling noise schedules, and guidance mechanisms (classifier guidance, classifier-free guidance, posterior sampling); (3) model architecture, including conditioning mechanisms (modulation, cross-attention, concatenation) and network backbones (WaveNet, U-Net, Transformers, State-Space Models). This modular design space has been systematically analyzed by frameworks like EDM karras2022elucidating and extends to recent flow matching approaches. The interchangeability of these components enables flexible experimentation across diverse audio generation tasks, from text-to-audio to voice cloning.
• Figure 2: Sampling trajectories for approximating a two-peak Dirac distribution, comparing 1D ODE solution curvatures for variance exploding (VE), variance preserving (VP), and EDM VE approaches. The horizontal axis $t$ represents the progression from initial noise ($t=1$) to the target distribution ($t=0$), normalized for better illustration. Local gradients for each curve are indicated with arrows.
• Figure 3: In-context conditioning: (a) sequence concatenation, where time tokens are embeddings output from MLP layers; (b) time aligned channel concatenation. In this example, we use a denoising objective; in flow matching, the output can be the "velocity".
• Figure 4: Adaptive layer norm modulation.
• Figure 5: Cross attention conditioning.