Training-Efficient Text-to-Music Generation with State-Space Modeling

TL;DR

The paper tackles openness and reproducibility in text-to-music generation by constraining models to a training- and data-efficient regime trained from public CC-licensed data. It replaces Transformer backbones with state-space models (SSMs) and experiments with three SSM-based LMs, including a two-stage SSM/diffusion design that uses coarse DAC tokens generated by an AR SSM LM followed by a latent diffusion model for fine tokens. Key findings show that SSMs, particularly Prefix SiMBA, converge faster and achieve competitive objective and subjective quality with only ~9% of the FLOPs and 2% of MusicGen-small’s data, and that the two-stage Prefix SiMBA/Diffusion model can match MusicGen-small’s performance under a limited budget. The work demonstrates substantial training efficiency gains and parameter efficiency, and promotes open, reproducible TTM research by releasing processed data, checkpoints, and code.

Abstract

Recent advances in text-to-music generation (TTM) have yielded high-quality results, but often at the cost of extensive compute and the use of large proprietary internal data. To improve the affordability and openness of TTM training, an open-source generative model backbone that is more training- and data-efficient is needed. In this paper, we constrain the number of trainable parameters in the generative model to match that of the MusicGen-small benchmark (with about 300M parameters), and replace its Transformer backbone with the emerging class of state-space models (SSMs). Specifically, we explore different SSM variants for sequence modeling, and compare a single-stage SSM-based design with a decomposable two-stage SSM/diffusion hybrid design. All proposed models are trained from scratch on a purely public dataset comprising 457 hours of CC-licensed music, ensuring full openness. Our experimental findings are three-fold. First, we show that SSMs exhibit superior training efficiency compared to the Transformer counterpart. Second, despite using only 9% of the FLOPs and 2% of the training data size compared to the MusicGen-small benchmark, our model achieves competitive performance in both objective metrics and subjective listening tests based on MusicCaps captions. Finally, our scaling-down experiment demonstrates that SSMs can maintain competitive performance relative to the Transformer baseline even at the same training budget (measured in iterations), when the model size is reduced to four times smaller. To facilitate the democratization of TTM research, the processed captions, model checkpoints, and source code are available on GitHub via the project page: https://lonian6.github.io/ssmttm/.

Figures (3)

• Figure 1: LM architectures. (Left): overall architecture, using the DAC codec kumar2023dac for the input RVQ audio tokens and Flan-T5 chung2024t5 for processing the input text. (Right) (a)--(d): the LM variants evaluated, including Mamba-2 dao2024mamba2, SiMBA patro2024SiMBA, and Transformer, using either prefix or cross-attention to condition the generation on the input text.
• Figure 2: Objective result of two selected LMs as a function of training steps. Performance is evaluated across training steps within the two-stage framework (Diffusion stage is frozen). Prefix SiMBA demonstrates superior training efficiency.
• Figure 3: Comparison of generated Mel-spectrograms from SiMBA-based and Transformer-based diffusion models using the same text prompt describing a jazz-reggae concert with multiple instruments. Sub-figures (a)-(d) illustrate the visual consistency and spectral detail across different architectures and training iterations.