ReactDance: Hierarchical Representation for High-Fidelity and Coherent Long-Form Reactive Dance Generation

TL;DR

A diffusion framework that operates on a novel hierarchical latent space to address spatiotemporal challenges in reactive dance generation, and substantially outperforms state-of-the-art methods in motion quality, long-term coherence, and sampling efficiency.

Abstract

Reactive dance generation (RDG), the task of generating a dance conditioned on a lead dancer's motion, holds significant promise for enhancing human-robot interaction and immersive digital entertainment. Despite progress in duet synchronization and motion-music alignment, two key challenges remain: generating fine-grained spatial interactions and ensuring long-term temporal coherence. In this work, we introduce \textbf{ReactDance}, a diffusion framework that operates on a novel hierarchical latent space to address these spatiotemporal challenges in RDG. First, for high-fidelity spatial expression and fine-grained control, we propose Hierarchical Finite Scalar Quantization (\textbf{HFSQ}). This multi-scale motion representation effectively disentangles coarse body posture from subtle limb dynamics, enabling independent and detailed control over both aspects through a layered guidance mechanism. Second, to efficiently generate long sequences with high temporal coherence, we propose Blockwise Local Context (\textbf{BLC}), a non-autoregressive sampling strategy. Departing from slow, frame-by-frame generation, BLC partitions the sequence into blocks and synthesizes them in parallel via periodic causal masking and positional encodings. Coherence across these blocks is ensured by a dense sliding-window training approach that enriches the representation with local temporal context. Extensive experiments show that ReactDance substantially outperforms state-of-the-art methods in motion quality, long-term coherence, and sampling efficiency. Project page: https://ripemangobox.github.io/ReactDance.

Figures (9)

• Figure 1: ReactDance generates high-fidelity, long-form reactive dance Conditioned on a leader and music, our model uses a hierarchical representation to refine motion from coarse (translucent) to fine-grained (solid). Its parallel sampling mechanism produces coherent sequences exceeding 2000 frames within 2 seconds, avoiding the quality degradation and slow speed of autoregressive sampling.
• Figure 2: Motion HFSQ overview. The proposed motion HFSQ learns to progressively encode motion sequence into a hierarchical representation $\mathcal{V} = \{ \hat{\bm{v}}_{g,r} \}_{g=1,r=1}^{G,R}$ and reconstructs motions via a grouped residual architecture. Building on FSQ, HFSQ eliminates codebook collapse while enhancing multi-scale motion expressiveness through grouped residual quantization, which integrates coarse-to-fine motion semantics.
• Figure 3: Progressive masking overview.
• Figure 4: ReactDance Pipeline Overview. Our ReactDance generates long, high-fidelity reactive dance sequences conditioned on leader motion and music. The core is a diffusion model that learns to denoise hierarchical HFSQ latents. Leader motion is injected via cross-attention, while music features are fused using a FiLM layer. For coherent generation of long sequences, our Blockwise Local Context (BLC) sampling strategy partitions the timeline into parallel blocks with aligned temporal contexts. Within each denoising step, Layer-Decoupled Classifier-Free Guidance (LDCFG) provides fine-grained control by applying independent guidance weights to each HFSQ scale. Finally, the denoised latents are decoded into a dance sequence coherently aligned with the input conditions.
• Figure 5: Qualitative comparison of reactive dance generation. Given the same leader motion, Duolando produces unnatural head rotations and GestureLSM shows uncoordinated interactions. InterGen's motion collapses into unrealistic jitter when generalizing beyond its training horizon. In contrast, our model generates a fluid and coherent reactive motion.