Exploring the Efficacy of Pre-trained Checkpoints in Text-to-Music Generation Task
Shangda Wu, Maosong Sun
TL;DR
This work tackles text-to-symbolic music generation, addressing the data scarcity in symbolic music by leveraging pre-trained NLP checkpoints to initialize a transformer-based seq2seq model. The authors introduce Textune, a large-scale dataset of 282,870 text–music pairs encoded in ABC notation, and systematically evaluate encoder-only and encoder–decoder initializations (BERT, GPT-2, BART) on this task. Results show that pre-trained checkpoints can significantly improve alignment with ground truth for some configurations (notably GPT-2 and BART-base) in terms of BLEU-N and EDS, but do not universally reduce validation loss and can reduce diversity or cause overfitting in some cases. The study highlights the critical role of data scale for achieving higher creativity and generalization in language–music models, suggesting that larger, more diverse text–music corpora are needed to unlock broader capabilities like melody style transfer and richer musical creativity.
Abstract
Benefiting from large-scale datasets and pre-trained models, the field of generative models has recently gained significant momentum. However, most datasets for symbolic music are very small, which potentially limits the performance of data-driven multimodal models. An intuitive solution to this problem is to leverage pre-trained models from other modalities (e.g., natural language) to improve the performance of symbolic music-related multimodal tasks. In this paper, we carry out the first study of generating complete and semantically consistent symbolic music scores from text descriptions, and explore the efficacy of using publicly available checkpoints (i.e., BERT, GPT-2, and BART) for natural language processing in the task of text-to-music generation. Our experimental results show that the improvement from using pre-trained checkpoints is statistically significant in terms of BLEU score and edit distance similarity. We analyse the capabilities and limitations of our model to better understand the potential of language-music models.