PF-D2M: A Pose-free Diffusion Model for Universal Dance-to-Music Generation
Jaekwon Im, Natalia Polouliakh, Taketo Akama
TL;DR
PF-D2M tackles universal dance-to-music generation by leveraging visual features from dance videos rather than relying solely on single-dancer pose cues. It employs a diffusion-based architecture (DiT) with a VAE latent for audio and conditioning from text (via T5) and Synchformer visual features, fused through AdaLN and learned timesteps. A three-stage progressive training strategy—text-to-audio initialization, video-to-audio alignment, and dance-to-music fine-tuning—mitigates data scarcity and improves generalization across real-world, multi-dancer, and non-human scenarios. Experimental results on AIST++ under the LORIS benchmark show state-of-the-art dance-music alignment and perceptual music quality, with subjective listening tests confirming robust performance in wild, diverse settings. This approach broadens the applicability of dance-to-music systems for choreographers and digital content creation by delivering high-quality, rhythmically coherent music aligned to complex dance videos.
Abstract
Dance-to-music generation aims to generate music that is aligned with dance movements. Existing approaches typically rely on body motion features extracted from a single human dancer and limited dance-to-music datasets, which restrict their performance and applicability to real-world scenarios involving multiple dancers and non-human dancers. In this paper, we propose PF-D2M, a universal diffusion-based dance-to-music generation model that incorporates visual features extracted from dance videos. PF-D2M is trained with a progressive training strategy that effectively addresses data scarcity and generalization challenges. Both objective and subjective evaluations show that PF-D2M achieves state-of-the-art performance in dance-music alignment and music quality.