Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Generalized Multi-Source Inference for Text Conditioned Music Diffusion Models

Emilian Postolache, Giorgio Mariani, Luca Cosmo, Emmanouil Benetos, Emanuele Rodolà

TL;DR

This paper proposes an inference procedure enabling the coherent generation of sources and accompaniments in arbitrary time-domain diffusion models conditioned on text embeddings and adapts the Dirac separator of MSDM to perform source separation.

Abstract

Multi-Source Diffusion Models (MSDM) allow for compositional musical generation tasks: generating a set of coherent sources, creating accompaniments, and performing source separation. Despite their versatility, they require estimating the joint distribution over the sources, necessitating pre-separated musical data, which is rarely available, and fixing the number and type of sources at training time. This paper generalizes MSDM to arbitrary time-domain diffusion models conditioned on text embeddings. These models do not require separated data as they are trained on mixtures, can parameterize an arbitrary number of sources, and allow for rich semantic control. We propose an inference procedure enabling the coherent generation of sources and accompaniments. Additionally, we adapt the Dirac separator of MSDM to perform source separation. We experiment with diffusion models trained on Slakh2100 and MTG-Jamendo, showcasing competitive generation and separation results in a relaxed data setting.

Generalized Multi-Source Inference for Text Conditioned Music Diffusion Models

TL;DR

This paper proposes an inference procedure enabling the coherent generation of sources and accompaniments in arbitrary time-domain diffusion models conditioned on text embeddings and adapts the Dirac separator of MSDM to perform source separation.

Abstract

Multi-Source Diffusion Models (MSDM) allow for compositional musical generation tasks: generating a set of coherent sources, creating accompaniments, and performing source separation. Despite their versatility, they require estimating the joint distribution over the sources, necessitating pre-separated musical data, which is rarely available, and fixing the number and type of sources at training time. This paper generalizes MSDM to arbitrary time-domain diffusion models conditioned on text embeddings. These models do not require separated data as they are trained on mixtures, can parameterize an arbitrary number of sources, and allow for rich semantic control. We propose an inference procedure enabling the coherent generation of sources and accompaniments. Additionally, we adapt the Dirac separator of MSDM to perform source separation. We experiment with diffusion models trained on Slakh2100 and MTG-Jamendo, showcasing competitive generation and separation results in a relaxed data setting.
Paper Structure (12 sections, 17 equations, 3 figures, 2 tables)

This paper contains 12 sections, 17 equations, 3 figures, 2 tables.

Table of Contents

  1. INTRODUCTION
  2. BACKGROUND
  3. Text-conditioned Score-based Diffusion Models
  4. Multi-Source Diffusion Models
  5. GENERALIZED MULTI-SOURCE DIFFUSION INFERENCE
  6. Total generation
  7. Partial generation
  8. Source separation
  9. EXPERIMENTAL SETUP
  10. EXPERIMENTAL RESULTS
  11. CONCLUSIONS
  12. ACKNOWLEDGEMENTS

Figures (3)

  • Figure 1: Diagram for unconditional generation procedure with GMSDI, sampling two coherent sources.
  • Figure 2: FAD (lower is better) between generated sources and Slakh100 test data (200 chunks, $\sim$12s each). Neg Prompt indicates the presence of negative prompting.
  • Figure 3: FAD (lower is better) results on total and partial generation, with respect to Slakh2100 test mixtures (200 chunks, $\sim$12s each).