Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Fast Timing-Conditioned Latent Audio Diffusion

Zach Evans, CJ Carr, Josiah Taylor, Scott H. Hawley, Jordi Pons

TL;DR

The paper presents Stable Audio, a latent-diffusion framework for fast, variable-length 44.1kHz stereo audio generation conditioned on text prompts and explicit timing signals. It combines a fully convolutional VAE, a CLAP-based text encoder, and a large diffusion U-Net with cross-attention for text and timing, enabling outputs up to 95 seconds in under 10 seconds of inference on an A100. The authors introduce specialized metrics for long-form, full-band stereo audio (FD_openl3, KL_past, CLAP_score) and demonstrate state-of-the-art performance on MusicCaps and AudioCaps, including structured music and stereo sound effects. They also show that their approach outperforms autoregressive and some diffusion baselines in audio quality and text alignment while offering a substantial speed advantage due to latent diffusion and efficient inference.

Abstract

Generating long-form 44.1kHz stereo audio from text prompts can be computationally demanding. Further, most previous works do not tackle that music and sound effects naturally vary in their duration. Our research focuses on the efficient generation of long-form, variable-length stereo music and sounds at 44.1kHz using text prompts with a generative model. Stable Audio is based on latent diffusion, with its latent defined by a fully-convolutional variational autoencoder. It is conditioned on text prompts as well as timing embeddings, allowing for fine control over both the content and length of the generated music and sounds. Stable Audio is capable of rendering stereo signals of up to 95 sec at 44.1kHz in 8 sec on an A100 GPU. Despite its compute efficiency and fast inference, it is one of the best in two public text-to-music and -audio benchmarks and, differently from state-of-the-art models, can generate music with structure and stereo sounds.

Fast Timing-Conditioned Latent Audio Diffusion

TL;DR

The paper presents Stable Audio, a latent-diffusion framework for fast, variable-length 44.1kHz stereo audio generation conditioned on text prompts and explicit timing signals. It combines a fully convolutional VAE, a CLAP-based text encoder, and a large diffusion U-Net with cross-attention for text and timing, enabling outputs up to 95 seconds in under 10 seconds of inference on an A100. The authors introduce specialized metrics for long-form, full-band stereo audio (FD_openl3, KL_past, CLAP_score) and demonstrate state-of-the-art performance on MusicCaps and AudioCaps, including structured music and stereo sound effects. They also show that their approach outperforms autoregressive and some diffusion baselines in audio quality and text alignment while offering a substantial speed advantage due to latent diffusion and efficient inference.

Abstract

Generating long-form 44.1kHz stereo audio from text prompts can be computationally demanding. Further, most previous works do not tackle that music and sound effects naturally vary in their duration. Our research focuses on the efficient generation of long-form, variable-length stereo music and sounds at 44.1kHz using text prompts with a generative model. Stable Audio is based on latent diffusion, with its latent defined by a fully-convolutional variational autoencoder. It is conditioned on text prompts as well as timing embeddings, allowing for fine control over both the content and length of the generated music and sounds. Stable Audio is capable of rendering stereo signals of up to 95 sec at 44.1kHz in 8 sec on an A100 GPU. Despite its compute efficiency and fast inference, it is one of the best in two public text-to-music and -audio benchmarks and, differently from state-of-the-art models, can generate music with structure and stereo sounds.
Paper Structure (31 sections, 7 figures, 4 tables)

This paper contains 31 sections, 7 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Architecture
  4. Variational autoencoder (VAE)
  5. Conditioning
  6. Diffusion model
  7. Inference
  8. Training
  9. Dataset
  10. Variational autoencoder (VAE)
  11. Text encoder
  12. Diffusion model
  13. Prompt preparation
  14. Methodology
  15. Quantitative metrics
  16. ...and 16 more sections

Figures (7)

  • Figure 1: Stable Audio. Blue: frozen pre-trained models. Green: parameters learnt during diffusion training. Purple: signals of interest.
  • Figure 2: Timing embeddings examples.Left: Audio file longer than training window. Right: Audio file shorter than training window.
  • Figure 3: Comparing the actual length (measured in the signal) against the expected length (provided by the timing conditioning).
  • Figure 4: Quality metrics vs Inference diffusion steps (trade-off).
  • Figure 5: Statistics of the MusicCaps original data.
  • ...and 2 more figures