Dancing Points: Synthesizing Ballroom Dancing with Three-Point Inputs

TL;DR

This work tackles real-time two-person ballroom dancing synthesis from sparse $3$-point VR inputs by treating the head and hands as a compact motion descriptor. It introduces a deterministic two-network pipeline: a tracking network that predicts full-body motion autoregressively from current pose and future $3$-point trajectories, and a non-autoregressive mapping network that forecasts future leader and follower $3$-point trajectories from the leader's past input. By leveraging the low-dimensional descriptor, the approach achieves stable rollouts and strong performance on both small ballroom datasets and large-scale LaFAN, while remaining computationally efficient on commodity hardware. The results show high-precision three-point tracking and plausible follower synthesis, offering a practical pathway for immersive, paired dancing in VR with data-efficient learning.

Abstract

Ballroom dancing is a structured yet expressive motion category. Its highly diverse movement and complex interactions between leader and follower dancers make the understanding and synthesis challenging. We demonstrate that the three-point trajectory available from a virtual reality (VR) device can effectively serve as a dancer's motion descriptor, simplifying the modeling and synthesis of interplay between dancers' full-body motions down to sparse trajectories. Thanks to the low dimensionality, we can employ an efficient MLP network to predict the follower's three-point trajectory directly from the leader's three-point input for certain types of ballroom dancing, addressing the challenge of modeling high-dimensional full-body interaction. It also prevents our method from overfitting thanks to its compact yet explicit representation. By leveraging the inherent structure of the movements and carefully planning the autoregressive procedure, we show a deterministic neural network is able to translate three-point trajectories into a virtual embodied avatar, which is typically considered under-constrained and requires generative models for common motions. In addition, we demonstrate this deterministic approach generalizes beyond small, structured datasets like ballroom dancing, and performs robustly on larger, more diverse datasets such as LaFAN. Our method provides a computationally- and data-efficient solution, opening new possibilities for immersive paired dancing applications. Code and pre-trained models for this paper are available at https://peizhuoli.github.io/dancing-points.

Figures (10)

• Figure 1: Our framework begins with the three-point input from the past 0.5 seconds of the leading dancer. The mapping network maps this input to the future three-point positions for both the leader and the follower. Using these predicted three-point positions, along with the pose predicted from the previous frame, the tracking network autoregressively predicts the future root trajectories and the future full-body motions.
• Figure 2: Overview of the tracking network. (a) During training, we use an auto-encoder to learn the latent space, and an estimator to predict the latent code $z_s$ that corresponds to the input $\textbf{X}$. (b) During inference, the encoder is discarded, reducing the tracking network to an MLP.
• Figure 3: Comparison of three-point tracking. Each row corresponds to a frame. MLP is unstable in the autoregressive rollout. Codebook matching (CBM) starke2024categorical has difficulty modeling highly dynamic motion due to the discrete latent space. Our model captures details and faithfully reflects the movement of the three-point input.
• Figure 4: Three-point tracking on LaFAN harvey2020robust dataset. Our prediction (blue) follows the three-point input, though it may not match the ground-truth (white) exactly due to dataset diversity.
• Figure 5: Comparison for follower dancer prediction. The predicted follower is in red. CAMDM's prediction chen2024taming predominantly relies on the prediction of previous frames, which leads to error accumulation and overfitting. Our model is responsive to the input signal and produces motion that naturally interacts with the leader.