Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

GACA-DiT: Diffusion-based Dance-to-Music Generation with Genre-Adaptive Rhythm and Context-Aware Alignment

Jinting Wang, Chenxing Li, Li Liu

TL;DR

GACA-DiT tackles the core problems of dance-to-music generation by introducing a genre-adaptive rhythm extraction module and a context-aware temporal alignment module within a diffusion-transformer framework. The GARE component captures fine-grained, genre-specific rhythmic cues using multi-scale wavelet features and spatial phase histograms, while CATA aligns the dense rhythm representations with downsampled music latents via learnable context queries. The model is trained as a conditional diffusion process conditioned on both the rhythm embedding and video features, producing music latents that are decoded into waveform with a VAE, achieving improved objective metrics and subjective quality over state-of-the-art methods on AIST++ and TikTok. These contributions enhance rhythmic consistency and temporal synchronization in cross-modal generation, with potential impact on automated scoring, virtual accompaniment, and user-generated content workflows.

Abstract

Dance-to-music (D2M) generation aims to automatically compose music that is rhythmically and temporally aligned with dance movements. Existing methods typically rely on coarse rhythm embeddings, such as global motion features or binarized joint-based rhythm values, which discard fine-grained motion cues and result in weak rhythmic alignment. Moreover, temporal mismatches introduced by feature downsampling further hinder precise synchronization between dance and music. To address these problems, we propose \textbf{GACA-DiT}, a diffusion transformer-based framework with two novel modules for rhythmically consistent and temporally aligned music generation. First, a \textbf{genre-adaptive rhythm extraction} module combines multi-scale temporal wavelet analysis and spatial phase histograms with adaptive joint weighting to capture fine-grained, genre-specific rhythm patterns. Second, a \textbf{context-aware temporal alignment} module resolves temporal mismatches using learnable context queries to align music latents with relevant dance rhythm features. Extensive experiments on the AIST++ and TikTok datasets demonstrate that GACA-DiT outperforms state-of-the-art methods in both objective metrics and human evaluation. Project page: https://beria-moon.github.io/GACA-DiT/.

GACA-DiT: Diffusion-based Dance-to-Music Generation with Genre-Adaptive Rhythm and Context-Aware Alignment

TL;DR

GACA-DiT tackles the core problems of dance-to-music generation by introducing a genre-adaptive rhythm extraction module and a context-aware temporal alignment module within a diffusion-transformer framework. The GARE component captures fine-grained, genre-specific rhythmic cues using multi-scale wavelet features and spatial phase histograms, while CATA aligns the dense rhythm representations with downsampled music latents via learnable context queries. The model is trained as a conditional diffusion process conditioned on both the rhythm embedding and video features, producing music latents that are decoded into waveform with a VAE, achieving improved objective metrics and subjective quality over state-of-the-art methods on AIST++ and TikTok. These contributions enhance rhythmic consistency and temporal synchronization in cross-modal generation, with potential impact on automated scoring, virtual accompaniment, and user-generated content workflows.

Abstract

Dance-to-music (D2M) generation aims to automatically compose music that is rhythmically and temporally aligned with dance movements. Existing methods typically rely on coarse rhythm embeddings, such as global motion features or binarized joint-based rhythm values, which discard fine-grained motion cues and result in weak rhythmic alignment. Moreover, temporal mismatches introduced by feature downsampling further hinder precise synchronization between dance and music. To address these problems, we propose \textbf{GACA-DiT}, a diffusion transformer-based framework with two novel modules for rhythmically consistent and temporally aligned music generation. First, a \textbf{genre-adaptive rhythm extraction} module combines multi-scale temporal wavelet analysis and spatial phase histograms with adaptive joint weighting to capture fine-grained, genre-specific rhythm patterns. Second, a \textbf{context-aware temporal alignment} module resolves temporal mismatches using learnable context queries to align music latents with relevant dance rhythm features. Extensive experiments on the AIST++ and TikTok datasets demonstrate that GACA-DiT outperforms state-of-the-art methods in both objective metrics and human evaluation. Project page: https://beria-moon.github.io/GACA-DiT/.

Paper Structure

This paper contains 12 sections, 6 equations, 3 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Related Work of D2M Methods
  3. Method
  4. Overview
  5. Genre-Adaptive Rhythm Extraction
  6. Context-aware Temporal Alignment
  7. Conditional Music Generation
  8. Experiment
  9. Experimental Setup
  10. Compared with SOTAs
  11. Ablation Analysis
  12. Conclusion

Figures (3)

  • Figure 1: Illustration of key problems in existing D2M methods: coarse rhythm embeddings and temporal misalignment.
  • Figure 2: Overview of the proposed GACA-DiT framework. It comprises a genre-adaptive rhythm extraction module, a semantic encoder, a context-aware temporal alignment module, and a Transformer-based diffusion model with music encoder–decoder.
  • Figure 3: Violin plots of MOS results for subjective evaluation.