MoMu-Diffusion: On Learning Long-Term Motion-Music Synchronization and Correspondence

TL;DR

Extensive experiments demonstrate that MoMu-Diffusion surpasses recent state-of-the-art methods both qualitatively and quantitatively, and can synthesize realistic, diverse, long-term, and beat-matched music or motion sequences.

Abstract

Motion-to-music and music-to-motion have been studied separately, each attracting substantial research interest within their respective domains. The interaction between human motion and music is a reflection of advanced human intelligence, and establishing a unified relationship between them is particularly important. However, to date, there has been no work that considers them jointly to explore the modality alignment within. To bridge this gap, we propose a novel framework, termed MoMu-Diffusion, for long-term and synchronous motion-music generation. Firstly, to mitigate the huge computational costs raised by long sequences, we propose a novel Bidirectional Contrastive Rhythmic Variational Auto-Encoder (BiCoR-VAE) that extracts the modality-aligned latent representations for both motion and music inputs. Subsequently, leveraging the aligned latent spaces, we introduce a multi-modal Transformer-based diffusion model and a cross-guidance sampling strategy to enable various generation tasks, including cross-modal, multi-modal, and variable-length generation. Extensive experiments demonstrate that MoMu-Diffusion surpasses recent state-of-the-art methods both qualitatively and quantitatively, and can synthesize realistic, diverse, long-term, and beat-matched music or motion sequences. The generated samples and codes are available at https://momu-diffusion.github.io/

Figures (12)

• Figure 1: The pipeline of MoMu-Diffusion. MoMu-Diffusion integrates the alignment of motion and music through the novel Bidirectional Contrastive Rhythmic Auto-Encoder (BiCoR-VAE). Leveraging the aligned latent space, MoMu-Diffusion facilitates both cross-modal and multi-modal generations.
• Figure 2: An overview of the proposed MoMu-Diffusion framework. MoMu-Diffusion contains two integral components: a bidirectional contrastive rhythmic Variational Autoencoder (BiCoR-VAE) designed to learn the aligned latent space, and a Transformer-based diffusion model responsible for sequence generation. This framework is adept at facilitating both cross-modal and multi-modal joint generations, offering a robust approach to the integrated synthesis of motion and music.
• Figure 2: Motion-to-music with beat-matching metrics.
• Figure 3: Motion-to-music with generation quality metrics: FAD$\downarrow$ and Diversity$\uparrow$.
• Figure 4: Example of beat matching on the motion-to-music generation. The red dashes indicate the extracted musical beats. The red arrow points to the video frame at that particular moment.