Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

MusicFlow: Cascaded Flow Matching for Text Guided Music Generation

K R Prajwal, Bowen Shi, Matthew Lee, Apoorv Vyas, Andros Tjandra, Mahi Luthra, Baishan Guo, Huiyu Wang, Triantafyllos Afouras, David Kant, Wei-Ning Hsu

TL;DR

MusicFlow, a cascaded text-to-music generation model based on flow matching based on self-supervised representations to bridge between text descriptions and music audios, is introduced and two flow matching networks are constructed to model the conditional distribution of semantic and acoustic features.

Abstract

We introduce MusicFlow, a cascaded text-to-music generation model based on flow matching. Based on self-supervised representations to bridge between text descriptions and music audios, we construct two flow matching networks to model the conditional distribution of semantic and acoustic features. Additionally, we leverage masked prediction as the training objective, enabling the model to generalize to other tasks such as music infilling and continuation in a zero-shot manner. Experiments on MusicCaps reveal that the music generated by MusicFlow exhibits superior quality and text coherence despite being over $2\sim5$ times smaller and requiring $5$ times fewer iterative steps. Simultaneously, the model can perform other music generation tasks and achieves competitive performance in music infilling and continuation. Our code and model will be publicly available.

MusicFlow: Cascaded Flow Matching for Text Guided Music Generation

TL;DR

MusicFlow, a cascaded text-to-music generation model based on flow matching based on self-supervised representations to bridge between text descriptions and music audios, is introduced and two flow matching networks are constructed to model the conditional distribution of semantic and acoustic features.

Abstract

We introduce MusicFlow, a cascaded text-to-music generation model based on flow matching. Based on self-supervised representations to bridge between text descriptions and music audios, we construct two flow matching networks to model the conditional distribution of semantic and acoustic features. Additionally, we leverage masked prediction as the training objective, enabling the model to generalize to other tasks such as music infilling and continuation in a zero-shot manner. Experiments on MusicCaps reveal that the music generated by MusicFlow exhibits superior quality and text coherence despite being over times smaller and requiring times fewer iterative steps. Simultaneously, the model can perform other music generation tasks and achieves competitive performance in music infilling and continuation. Our code and model will be publicly available.

Paper Structure

This paper contains 20 sections, 2 equations, 3 figures, 8 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Method
  4. Background: Flow matching
  5. Problem Formulation
  6. A Cascaded Flow-matching Approach
  7. Stage 1: Music Semantic Flow Matching from Text
  8. Stage 2: Music Acoustic Flow Matching from text and semantics
  9. Classifier-free guidance
  10. Experiments
  11. Experimental Setup
  12. Main Results
  13. Infilling and Continuation
  14. Ablation Study
  15. Single-stage vs. Two-stage Model
  16. ...and 5 more sections

Figures (3)

  • Figure 1: MusicFlow Diagram. Note the acoustic encoder, acoustic decoder and semantic encoder are pre-trained and frozen during generative model training. For text-to-music generation (i.e., 100% masking), both acoustic and semantic encoder are discarded in inference.
  • Figure 2: Pairwise comparison between MusicFlow, AudioLDM2, MusicGen and ground-truth
  • Figure 3: Comparison between MusicFlow and prior works in FAD-NFE in terms of inference efficiency.