A Survey on Music Generation from Single-Modal, Cross-Modal, and Multi-Modal Perspectives

TL;DR

This survey categorizes music generation through the lens of modality, tracing the shift from single-modal to cross- and multi-modal guidance across audio, symbolic, text, image, and video representations. It synthesizes architectures, datasets, and evaluation methods, highlighting how cross-modal and multi-modal signals improve controllability and realism while exposing data scarcity and evaluation gaps. The review documents a spectrum of techniques—from diffusion and transformer-based models to specialized symbolic encoders and cross-modal embeddings—demonstrating substantial progress yet underscoring continuing challenges in creativity, efficiency, and unified assessment. The work provides a roadmap toward large-scale, multi-modal music systems with robust alignment, richer datasets, and standardized evaluation to enable practical deployment in creative and industrial contexts.

Abstract

Multi-modal music generation, using multiple modalities like text, images, and video alongside musical scores and audio as guidance, is an emerging research area with broad applications. This paper reviews this field, categorizing music generation systems from the perspective of modalities. The review covers modality representation, multi-modal data alignment, and their utilization to guide music generation. Current datasets and evaluation methods are also discussed. Key challenges in this area include effective multi-modal integration, large-scale comprehensive datasets, and systematic evaluation methods. Finally, an outlook on future research directions is provided, focusing on creativity, efficiency, multi-modal alignment, and evaluation.

Figures (1)

• Figure 1: Overview of Multi-Modal Music Generation