Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Long-form music generation with latent diffusion

Zach Evans, Julian D. Parker, CJ Carr, Zack Zukowski, Josiah Taylor, Jordi Pons

TL;DR

Problem: existing text-conditioned music models struggle to generate coherent long-form tracks with structural continuity. Approach: a three-component latent-diffusion pipeline operating on a highly downsampled latent, enabling up to 4m45s of music conditioned by CLAP text prompts. Contributions: autoencoder with high temporal downsampling, diffusion-transformer in latent space, and extensive quantitative and qualitative evaluation showing state-of-the-art audio quality and long-range structure, plus memorization analysis and creative capabilities. Impact: enables controllable, long-form, high-quality music generation with potential applications in music production and media.

Abstract

Audio-based generative models for music have seen great strides recently, but so far have not managed to produce full-length music tracks with coherent musical structure from text prompts. We show that by training a generative model on long temporal contexts it is possible to produce long-form music of up to 4m45s. Our model consists of a diffusion-transformer operating on a highly downsampled continuous latent representation (latent rate of 21.5Hz). It obtains state-of-the-art generations according to metrics on audio quality and prompt alignment, and subjective tests reveal that it produces full-length music with coherent structure.

Long-form music generation with latent diffusion

TL;DR

Problem: existing text-conditioned music models struggle to generate coherent long-form tracks with structural continuity. Approach: a three-component latent-diffusion pipeline operating on a highly downsampled latent, enabling up to 4m45s of music conditioned by CLAP text prompts. Contributions: autoencoder with high temporal downsampling, diffusion-transformer in latent space, and extensive quantitative and qualitative evaluation showing state-of-the-art audio quality and long-range structure, plus memorization analysis and creative capabilities. Impact: enables controllable, long-form, high-quality music generation with potential applications in music production and media.

Abstract

Audio-based generative models for music have seen great strides recently, but so far have not managed to produce full-length music tracks with coherent musical structure from text prompts. We show that by training a generative model on long temporal contexts it is possible to produce long-form music of up to 4m45s. Our model consists of a diffusion-transformer operating on a highly downsampled continuous latent representation (latent rate of 21.5Hz). It obtains state-of-the-art generations according to metrics on audio quality and prompt alignment, and subjective tests reveal that it produces full-length music with coherent structure.
Paper Structure (17 sections, 4 figures, 5 tables)

This paper contains 17 sections, 4 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Latent diffusion architecture
  3. Autoencoder
  4. Diffusion-transformer (DiT)
  5. Variable-length music generation
  6. CLAP text encoder
  7. Training Setup
  8. Training data and prompt preparation
  9. Experiments
  10. Quantitative evaluation
  11. Qualitative evaluation
  12. Autoencoder evaluation
  13. Musical structure analysis
  14. Memorization analysis
  15. Additional creative capabilities
  16. ...and 2 more sections

Figures (4)

  • Figure 1: Cumulative histogram showing the proportion of music that is less than a particular length, for a representative sample of popular music$^1$. Dotted lines: proportion associated with the max generation length of our model (285s) and of previous models (90s). The vertical axis is warped with a power law for greater readability.
  • Figure 2: Architecture of the diffusion-transformer (DiT). Cross-attention includes timing and text conditioning. Prepend conditioning includes timing conditioning and also the signal conditioning on the current timestep of the diffusion process.
  • Figure 3: Architecture of the autoencoder.
  • Figure 4: Each column shows the SSMs of different genres (left to right): rock, pop, jazz, hip-hop, and classical.