DanceAnyWay: Synthesizing Beat-Guided 3D Dances with Randomized Temporal Contrastive Learning

TL;DR

DanceAnyWay is a generative learning method to synthesize beat-guided dances of 3D human characters synchronized with music that learns to disentangle the dance movements at the beat frames from the dance movements at all the remaining frames by operating at two hierarchical levels.

Abstract

We present DanceAnyWay, a generative learning method to synthesize beat-guided dances of 3D human characters synchronized with music. Our method learns to disentangle the dance movements at the beat frames from the dance movements at all the remaining frames by operating at two hierarchical levels. At the coarser "beat" level, it encodes the rhythm, pitch, and melody information of the input music via dedicated feature representations only at the beat frames. It leverages them to synthesize the beat poses of the target dances using a sequence-to-sequence learning framework. At the finer "repletion" level, our method encodes similar rhythm, pitch, and melody information from all the frames of the input music via dedicated feature representations. It generates the full dance sequences by combining the synthesized beat and repletion poses and enforcing plausibility through an adversarial learning framework. Our training paradigm also enforces fine-grained diversity in the synthesized dances through a randomized temporal contrastive loss, which ensures different segments of the dance sequences have different movements and avoids motion freezing or collapsing to repetitive movements. We evaluate the performance of our approach through extensive experiments on the benchmark AIST++ dataset and observe improvements of about 7%-12% in motion quality metrics and 1.5%-4% in motion diversity metrics over the current baselines, respectively. We also conducted a user study to evaluate the visual quality of our synthesized dances. We note that, on average, the samples generated by our method were about 9-48% more preferred by the participants and had a 4-27% better five-point Likert-scale score over the best available current baseline in terms of motion quality and synchronization. Our source code and project page are available at https://github.com/aneeshbhattacharya/DanceAnyWay.

Figures (7)

• Figure 1: DanceAnyWay: A two-stage hierarchical network that can generate beat-aligned and diverse, fine-grained 3D dances given audio. We render our results with Mixamo characters.
• Figure 2: DanceAnyWay Network Architecture. DanceAnyWay consists of two stages, Beat Pose Synthesis (BPS) and Repletion Pose Synthesis (RPS), trained one after the other. BPS (top row, left) has a predictor architecture to generate the coarse beat poses, and RPS (bottom row) has a generative adversarial architecture to generate all the remaining poses with fine-grained detail, followed by a seq-to-seq trajectory predictor for the global root translations. To train our RPS, we propose an additional randomized temporal contrastive loss (top row, right) to enforce motion diversity. For completeness, we also expand our MFCC and Chroma encoders (top row, middle), which have the same architecture but different layer sizes.
• Figure 3: $t$-SNE Plot of Samples from RPS Latent Decoder Space. Distribution of the features for the $m$-length segments in $\mathcal{Z}_{\mathcal{R}}$, for 100 random samples (each represented with a different color) in AIST++ aist++, after training with (right) and without (left) our RTC loss. Clustering all the sample segments using the RTC loss is necessary to generate diverse motions.
• Figure 4: Beat Alignment. Kinetic velocities over time for one ground truth (GT) motion and corresponding generative results. Our method has more peaks and valleys at the beat frames, indicating more alignment with the audio.
• Figure 5: Visualizations on AIST++ aist++. Sampled frames in a left-to-right sequence for one test sample. Our generated samples are better aligned with beats, more diverse, and have more plausible fine-grained details.