Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Mustango: Toward Controllable Text-to-Music Generation

Jan Melechovsky, Zixun Guo, Deepanway Ghosal, Navonil Majumder, Dorien Herremans, Soujanya Poria

TL;DR

Mustango tackles the limited controllability of diffusion-based text-to-music by introducing MuNet, a music-domain-knowledge-informed UNet that conditions on tempo, key, and chord progressions in addition to text. It constructs MusicBench, a 53k-sample dataset augmented with music-theory attributes via MIR tools and targeted audio transformations, to train a controllable latent-diffusion model. Empirical results show Mustango achieves state-of-the-art musicality and markedly improved controllability over baselines like Tango, MusicGen, and AudioLDM2, with strong chord and tempo control demonstrated in objective and subjective evaluations. The work provides public access to MusicBench and Mustango, highlighting practical implications for music production and AI-assisted composition while acknowledging Western-centric limitations and scope for longer-form generation.

Abstract

The quality of the text-to-music models has reached new heights due to recent advancements in diffusion models. The controllability of various musical aspects, however, has barely been explored. In this paper, we propose Mustango: a music-domain-knowledge-inspired text-to-music system based on diffusion. Mustango aims to control the generated music, not only with general text captions, but with more rich captions that can include specific instructions related to chords, beats, tempo, and key. At the core of Mustango is MuNet, a Music-Domain-Knowledge-Informed UNet guidance module that steers the generated music to include the music-specific conditions, which we predict from the text prompt, as well as the general text embedding, during the reverse diffusion process. To overcome the limited availability of open datasets of music with text captions, we propose a novel data augmentation method that includes altering the harmonic, rhythmic, and dynamic aspects of music audio and using state-of-the-art Music Information Retrieval methods to extract the music features which will then be appended to the existing descriptions in text format. We release the resulting MusicBench dataset which contains over 52K instances and includes music-theory-based descriptions in the caption text. Through extensive experiments, we show that the quality of the music generated by Mustango is state-of-the-art, and the controllability through music-specific text prompts greatly outperforms other models such as MusicGen and AudioLDM2.

Mustango: Toward Controllable Text-to-Music Generation

TL;DR

Mustango tackles the limited controllability of diffusion-based text-to-music by introducing MuNet, a music-domain-knowledge-informed UNet that conditions on tempo, key, and chord progressions in addition to text. It constructs MusicBench, a 53k-sample dataset augmented with music-theory attributes via MIR tools and targeted audio transformations, to train a controllable latent-diffusion model. Empirical results show Mustango achieves state-of-the-art musicality and markedly improved controllability over baselines like Tango, MusicGen, and AudioLDM2, with strong chord and tempo control demonstrated in objective and subjective evaluations. The work provides public access to MusicBench and Mustango, highlighting practical implications for music production and AI-assisted composition while acknowledging Western-centric limitations and scope for longer-form generation.

Abstract

The quality of the text-to-music models has reached new heights due to recent advancements in diffusion models. The controllability of various musical aspects, however, has barely been explored. In this paper, we propose Mustango: a music-domain-knowledge-inspired text-to-music system based on diffusion. Mustango aims to control the generated music, not only with general text captions, but with more rich captions that can include specific instructions related to chords, beats, tempo, and key. At the core of Mustango is MuNet, a Music-Domain-Knowledge-Informed UNet guidance module that steers the generated music to include the music-specific conditions, which we predict from the text prompt, as well as the general text embedding, during the reverse diffusion process. To overcome the limited availability of open datasets of music with text captions, we propose a novel data augmentation method that includes altering the harmonic, rhythmic, and dynamic aspects of music audio and using state-of-the-art Music Information Retrieval methods to extract the music features which will then be appended to the existing descriptions in text format. We release the resulting MusicBench dataset which contains over 52K instances and includes music-theory-based descriptions in the caption text. Through extensive experiments, we show that the quality of the music generated by Mustango is state-of-the-art, and the controllability through music-specific text prompts greatly outperforms other models such as MusicGen and AudioLDM2.
Paper Structure (39 sections, 5 equations, 15 figures, 7 tables)

This paper contains 39 sections, 5 equations, 15 figures, 7 tables.

Table of Contents

  1. Introduction
  2. Dataset Creation
  3. Feature Extraction and Description Enrichment
  4. Augmentation and Music Diversification
  5. MusicBench
  6. Mustango
  7. Latent Diffusion Model (LDM)
  8. MuNet
  9. Inference
  10. Experiments
  11. Baselines and Mustango Variants
  12. Training and Additional Evaluation Set
  13. Inference Settings and Time
  14. Objective Evaluation Methodology
  15. Audio Quality Estimation
  16. ...and 24 more sections

Figures (15)

  • Figure 1: Composition of MusicBench dataset.
  • Figure 2: Depiction of our proposed Mustango model. Beats and chords are inferred from the caption when they are not provided as input.
  • Figure 3: Mel-spectrogram of a blues sample generated by Tango trained on MusicBench.
  • Figure 4: Mel-spectrogram of a blues sample generated by Mustango with vertical lines showing perceived chord starts.
  • Figure 5: Question interface used for the general listening test.
  • ...and 10 more figures