Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

ERNIE-Music: Text-to-Waveform Music Generation with Diffusion Models

Pengfei Zhu, Chao Pang, Yekun Chai, Lei Li, Shuohuan Wang, Yu Sun, Hao Tian, Hua Wu

TL;DR

ERNIE-Music addresses the challenge of directly generating music waveforms from unrestricted text prompts by employing a diffusion-based conditional model conditioned on free-form text. The approach uses a UNet backbone with a text encoder and a text-aware fusion mechanism to integrate linguistic prompts into waveform synthesis, and it compares end-to-end text conditioning against predefined music tags. A large web-derived Web Music with Text dataset is curated to support training despite scarce parallel data, and human evaluations demonstrate improved text-music relevance and audio quality over baselines. The work highlights the potential of waveform-based text-to-music generation and discusses limitations such as fixed sequence length and generation speed, pointing to future extensions to longer pieces and vocal content.

Abstract

In recent years, the burgeoning interest in diffusion models has led to significant advances in image and speech generation. Nevertheless, the direct synthesis of music waveforms from unrestricted textual prompts remains a relatively underexplored domain. In response to this lacuna, this paper introduces a pioneering contribution in the form of a text-to-waveform music generation model, underpinned by the utilization of diffusion models. Our methodology hinges on the innovative incorporation of free-form textual prompts as conditional factors to guide the waveform generation process within the diffusion model framework. Addressing the challenge of limited text-music parallel data, we undertake the creation of a dataset by harnessing web resources, a task facilitated by weak supervision techniques. Furthermore, a rigorous empirical inquiry is undertaken to contrast the efficacy of two distinct prompt formats for text conditioning, namely, music tags and unconstrained textual descriptions. The outcomes of this comparative analysis affirm the superior performance of our proposed model in terms of enhancing text-music relevance. Finally, our work culminates in a demonstrative exhibition of the excellent capabilities of our model in text-to-music generation. We further demonstrate that our generated music in the waveform domain outperforms previous works by a large margin in terms of diversity, quality, and text-music relevance.

ERNIE-Music: Text-to-Waveform Music Generation with Diffusion Models

TL;DR

ERNIE-Music addresses the challenge of directly generating music waveforms from unrestricted text prompts by employing a diffusion-based conditional model conditioned on free-form text. The approach uses a UNet backbone with a text encoder and a text-aware fusion mechanism to integrate linguistic prompts into waveform synthesis, and it compares end-to-end text conditioning against predefined music tags. A large web-derived Web Music with Text dataset is curated to support training despite scarce parallel data, and human evaluations demonstrate improved text-music relevance and audio quality over baselines. The work highlights the potential of waveform-based text-to-music generation and discusses limitations such as fixed sequence length and generation speed, pointing to future extensions to longer pieces and vocal content.

Abstract

In recent years, the burgeoning interest in diffusion models has led to significant advances in image and speech generation. Nevertheless, the direct synthesis of music waveforms from unrestricted textual prompts remains a relatively underexplored domain. In response to this lacuna, this paper introduces a pioneering contribution in the form of a text-to-waveform music generation model, underpinned by the utilization of diffusion models. Our methodology hinges on the innovative incorporation of free-form textual prompts as conditional factors to guide the waveform generation process within the diffusion model framework. Addressing the challenge of limited text-music parallel data, we undertake the creation of a dataset by harnessing web resources, a task facilitated by weak supervision techniques. Furthermore, a rigorous empirical inquiry is undertaken to contrast the efficacy of two distinct prompt formats for text conditioning, namely, music tags and unconstrained textual descriptions. The outcomes of this comparative analysis affirm the superior performance of our proposed model in terms of enhancing text-music relevance. Finally, our work culminates in a demonstrative exhibition of the excellent capabilities of our model in text-to-music generation. We further demonstrate that our generated music in the waveform domain outperforms previous works by a large margin in terms of diversity, quality, and text-music relevance.
Paper Structure (23 sections, 10 equations, 3 figures, 7 tables, 1 algorithm)

This paper contains 23 sections, 10 equations, 3 figures, 7 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. Controllable Music Generation
  4. Diffusion Models
  5. Method
  6. Unconditional Diffusion Model
  7. Conditional Diffusion Model
  8. Model Architecture
  9. Training
  10. Experiments
  11. Implementation Details
  12. Dataset
  13. Evaluation Metric
  14. Compared Methods
  15. Results
  16. ...and 8 more sections

Figures (3)

  • Figure 1: The overall architecture of text-to-music generation training. The text is input to the text encoder to obtain the sequence representation $S$, then $S$ and the sampled music waveform (noise added) $z_{t}$ are input to the UNet to obtain the estimated volocity $\hat{v}_t$, finally we calculate the L2 loss between $\hat{v}_t$ and the real volocity $v_t$. For the input text, the original Chinese is "钢琴旋律的弦音，轻轻地、温柔地倾诉心中的遐想、心中的爱恋".
  • Figure 2: The spectrogram and waveform of generated music examples. The model can generate diverse music, including smoothing and cadenced (a, c) and fast-paced (b, d) rhythms. Text of (a, c): The piano piece is light and comfortable yet deeply affectionate. Text of (b, d): A passionate, fast-paced guitar piece.
  • Figure 3: The MSE results on the test set for two implementations of the fusing operation.