Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Diffusion based Text-to-Music Generation with Global and Local Text based Conditioning

Jisi Zhang, Pablo Peso Parada, Md Asif Jalal, Karthikeyan Saravanan

TL;DR

This work tackles diffusion-based text-to-music generation by conditioning the UNet with both global and local text representations. It introduces a unified framework where a global embedding $G_y$ is delivered via FiLM and a local embedding $F_y$ is supplied through cross-attention, with the added capability to derive global text representations from local embeddings using mean pooling or self-attention pooling. Key contributions include an analysis of multiple text encoders for global conditioning, a dual-conditioning mechanism that injects global and local cues at different UNet levels without extra fusion modules, and an empirical demonstration that mean pooling can yield competitive results while reducing parameters. Findings show that dual-encoder configurations achieve strong KL (text adherence) scores, while mean pooling from a single encoder can substantially improve FAD (generation quality) with minimal parameter overhead, highlighting a practical path toward compact, high-quality TTM systems.

Abstract

Diffusion based Text-To-Music (TTM) models generate music corresponding to text descriptions. Typically UNet based diffusion models condition on text embeddings generated from a pre-trained large language model or from a cross-modality audio-language representation model. This work proposes a diffusion based TTM, in which the UNet is conditioned on both (i) a uni-modal language model (e.g., T5) via cross-attention and (ii) a cross-modal audio-language representation model (e.g., CLAP) via Feature-wise Linear Modulation (FiLM). The diffusion model is trained to exploit both a local text representation from the T5 and a global representation from the CLAP. Furthermore, we propose modifications that extract both global and local representations from the T5 through pooling mechanisms that we call mean pooling and self-attention pooling. This approach mitigates the need for an additional encoder (e.g., CLAP) to extract a global representation, thereby reducing the number of model parameters. Our results show that incorporating the CLAP global embeddings to the T5 local embeddings enhances text adherence (KL=1.47) compared to a baseline model solely relying on the T5 local embeddings (KL=1.54). Alternatively, extracting global text embeddings directly from the T5 local embeddings through the proposed mean pooling approach yields superior generation quality (FAD=1.89) while exhibiting marginally inferior text adherence (KL=1.51) against the model conditioned on both CLAP and T5 text embeddings (FAD=1.94 and KL=1.47). Our proposed solution is not only efficient but also compact in terms of the number of parameters required.

Diffusion based Text-to-Music Generation with Global and Local Text based Conditioning

TL;DR

This work tackles diffusion-based text-to-music generation by conditioning the UNet with both global and local text representations. It introduces a unified framework where a global embedding is delivered via FiLM and a local embedding is supplied through cross-attention, with the added capability to derive global text representations from local embeddings using mean pooling or self-attention pooling. Key contributions include an analysis of multiple text encoders for global conditioning, a dual-conditioning mechanism that injects global and local cues at different UNet levels without extra fusion modules, and an empirical demonstration that mean pooling can yield competitive results while reducing parameters. Findings show that dual-encoder configurations achieve strong KL (text adherence) scores, while mean pooling from a single encoder can substantially improve FAD (generation quality) with minimal parameter overhead, highlighting a practical path toward compact, high-quality TTM systems.

Abstract

Diffusion based Text-To-Music (TTM) models generate music corresponding to text descriptions. Typically UNet based diffusion models condition on text embeddings generated from a pre-trained large language model or from a cross-modality audio-language representation model. This work proposes a diffusion based TTM, in which the UNet is conditioned on both (i) a uni-modal language model (e.g., T5) via cross-attention and (ii) a cross-modal audio-language representation model (e.g., CLAP) via Feature-wise Linear Modulation (FiLM). The diffusion model is trained to exploit both a local text representation from the T5 and a global representation from the CLAP. Furthermore, we propose modifications that extract both global and local representations from the T5 through pooling mechanisms that we call mean pooling and self-attention pooling. This approach mitigates the need for an additional encoder (e.g., CLAP) to extract a global representation, thereby reducing the number of model parameters. Our results show that incorporating the CLAP global embeddings to the T5 local embeddings enhances text adherence (KL=1.47) compared to a baseline model solely relying on the T5 local embeddings (KL=1.54). Alternatively, extracting global text embeddings directly from the T5 local embeddings through the proposed mean pooling approach yields superior generation quality (FAD=1.89) while exhibiting marginally inferior text adherence (KL=1.51) against the model conditioned on both CLAP and T5 text embeddings (FAD=1.94 and KL=1.47). Our proposed solution is not only efficient but also compact in terms of the number of parameters required.
Paper Structure (11 sections, 6 equations, 1 figure, 3 tables)

This paper contains 11 sections, 6 equations, 1 figure, 3 tables.

Table of Contents

  1. Introduction
  2. Method
  3. Diffusion based generative model
  4. Conditioning models
  5. Multiple text conditioning models
  6. Experiment Setup
  7. Data
  8. Model configuration
  9. Results and Analysis
  10. Ablation study
  11. Conclusions

Figures (1)

  • Figure 1: Overview of our text-to-music diffusion model architecture conditioned over multiple text encoders. (a) A continuous latent space $z$ is created from a VAE model. (b) A latent diffusion model is conditioned on text embeddings to generate the audio latent $z$ from a noisy version or white noise $z_T$ in T sampling steps. The CLAP audio encoder is only applicable when a CLAP model is used as a conditioner during training. K, V, and Q indicate the key, value, and query respectively in the attention mechanism.