Multimodal Music Generation with Explicit Bridges and Retrieval Augmentation

TL;DR

This work tackles data scarcity and weak cross-modal alignment in multimodal music generation by introducing Visuals Music Bridge (VMB), which uses explicit text and music bridges to guide generation. The Multimodal Music Description Model converts visuals into textual descriptions (text bridge), while a Dual-track Music Retrieval module provides broad and targeted music references (music bridge); an Explicitly Conditioned Music Generation framework then fuses these bridges via a diffusion Transformer, employing Music ControlFormer and a Stylization Module for controllability. Across video-, image-, and text-to-music tasks, plus controllable generation and description automation, VMB achieves superior objective metrics ($KL_{passt}$, $FD_{openl3}$, CLAPScore, IB) and strong subjective alignment, while offering fast inference and fine-grained control. The approach sets a new benchmark for interpretable, customizable multimodal music generation with broad multimedia applicability and potential societal impact, highlighting avenues for dataset expansion and theory-informed improvements.

Abstract

Multimodal music generation aims to produce music from diverse input modalities, including text, videos, and images. Existing methods use a common embedding space for multimodal fusion. Despite their effectiveness in other modalities, their application in multimodal music generation faces challenges of data scarcity, weak cross-modal alignment, and limited controllability. This paper addresses these issues by using explicit bridges of text and music for multimodal alignment. We introduce a novel method named Visuals Music Bridge (VMB). Specifically, a Multimodal Music Description Model converts visual inputs into detailed textual descriptions to provide the text bridge; a Dual-track Music Retrieval module that combines broad and targeted retrieval strategies to provide the music bridge and enable user control. Finally, we design an Explicitly Conditioned Music Generation framework to generate music based on the two bridges. We conduct experiments on video-to-music, image-to-music, text-to-music, and controllable music generation tasks, along with experiments on controllability. The results demonstrate that VMB significantly enhances music quality, modality, and customization alignment compared to previous methods. VMB sets a new standard for interpretable and expressive multimodal music generation with applications in various multimedia fields. Demos and code are available at https://github.com/wbs2788/VMB.

Figures (14)

• Figure 1: Overview of the VMB framework. We employ text and music as two explicit bridges for multimodal music generation. Text-form music description is obtained with the Multimodal Music Description model. Reference music is retrieved with the Dual-track Music Retrieval module. The two bridges are fed into the Explicitly Conditioned Music Generation module to generate output music.
• Figure 2: Pipeline of the Multimodal Music Description Model (MMDM). This process starts with the collection of music videos, followed by automated tagging to refine audio annotations using CLAP embedding similarities. Metadata and thematic descriptions are synthesized by the Llama 3.1 model to create training targets. The training utilizes LoRA fine-tuning in the MMDM to transform multimodal inputs into targeted music descriptions that align with the visual content's themes.
• Figure 3: Framework of Dual-track Music Retrieval and Explicitly Conditioned Music Generation. The left part illustrates the Dual-track Music Retrieval process, which leverages our multimodal dataset to perform both broad and targeted retrieval. The right part shows the Explicitly Conditioned Music Generation pathway, where music is generated through a ControlFormer block integrating embeddings from selected music bridge, text bridge, and noisy inputs.
• Figure 4: Distribution of PAM Scores across the raw training dataset.
• Figure 5: Histogram of music duration in the training dataset.