Listen to Rhythm, Choose Movements: Autoregressive Multimodal Dance Generation via Diffusion and Mamba with Decoupled Dance Dataset

TL;DR

The paper addresses the challenge of controllable, coherent, long-range dance generation by introducing LRCM, a diffusion-based framework conditioned on audio rhythm and textual semantics within a semantic-decoupled Motorica Dance dataset. It integrates audio-latent Conformers and text-latent Cross-Conformers, along with a Motion Temporal Mamba Module (MTMM) to enable autoregressive long-sequence synthesis. The approach achieves state-of-the-art motion quality and diversity on the decoupled dataset, with competitive rhythm alignment and improved autoregressive transitions, while offering detailed ablations and robust qualitative results. This work demonstrates the practical potential of multimodal conditioning and SSM-based temporal modeling for extended, controllable dance generation, and provides code, dataset, and pretrained models for public use.

Abstract

Advances in generative models and sequence learning have greatly promoted research in dance motion generation, yet current methods still suffer from coarse semantic control and poor coherence in long sequences. In this work, we present Listen to Rhythm, Choose Movements (LRCM), a multimodal-guided diffusion framework supporting both diverse input modalities and autoregressive dance motion generation. We explore a feature decoupling paradigm for dance datasets and generalize it to the Motorica Dance dataset, separating motion capture data, audio rhythm, and professionally annotated global and local text descriptions. Our diffusion architecture integrates an audio-latent Conformer and a text-latent Cross-Conformer, and incorporates a Motion Temporal Mamba Module (MTMM) to enable smooth, long-duration autoregressive synthesis. Experimental results indicate that LRCM delivers strong performance in both functional capability and quantitative metrics, demonstrating notable potential in multimodal input scenarios and extended sequence generation. We will release the full codebase, dataset, and pretrained models publicly upon acceptance.

Figures (12)

• Figure 1: Example of decoupled textual modality annotations within the proposed paradigm.
• Figure 2: Overview of the proposed architecture. The main generation model is constructed using a DiT backbone. Text inputs are first processed through a fine-tuned LLM performing semantic tokenization, standardizing informal or non-standard prompts; the resulting tokens are then passed to a CLIP-based text encoder. Audio features are extracted via a high-level semantic encoder. The model stacks $N$ denoising residual blocks with residual and skip connections to integrate the encoded audio and text features for dance motion generation.
• Figure 3: Internal structure of the denoising residual block. Each block stacks $L$ computation layers following the Conformer architecture, applying audio-specific and text-specific processing respectively. Two conditional embedding modes are employed: Self-Attention for intra-modal feature refinement, and Cross-Attention for cross-modal integration. A Motion Temporal Mamba module is appended after the attention layers to capture long-range temporal dependencies.
• Figure 4: Motion Temporal Mamba Module (MTMM) architecture and process. Latent features from past motion memory and current motion are concatenated along the temporal axis, enriched with positional encoding, and processed by a bidirectional Mamba scan. The output is split to obtain the current segment's latent motion features for subsequent generation.
• Figure 5: Full LRCM model training strategy showing three-phase workflow and module freezing schedule.