Rhythm is a Dancer: Music-Driven Motion Synthesis with Global Structure

TL;DR

This work addresses the problem of generating long-term, music-driven dance motions that maintain global choreography while preserving local pose realism. It introduces a three-level hierarchical framework—pose, motif, and choreography—that jointly enforces beat synchronization, motif-consistent micro-movements, and a global dance signature, using a motion perceptual loss and adaptive style variation via Adaptive-Instance-Normalization. Key contributions include a novel motion perceptual loss, a choreography-level controller for global content, and demonstrations across salsa, modern, and folk styles with applications in choreography recreation and gap filling. The approach yields natural, diverse, and genre-consistent dances aligned to music, with potential impact on virtual performance, animation, and human-computer interaction scenarios.

Abstract

Synthesizing human motion with a global structure, such as a choreography, is a challenging task. Existing methods tend to concentrate on local smooth pose transitions and neglect the global context or the theme of the motion. In this work, we present a music-driven motion synthesis framework that generates long-term sequences of human motions which are synchronized with the input beats, and jointly form a global structure that respects a specific dance genre. In addition, our framework enables generation of diverse motions that are controlled by the content of the music, and not only by the beat. Our music-driven dance synthesis framework is a hierarchical system that consists of three levels: pose, motif, and choreography. The pose level consists of an LSTM component that generates temporally coherent sequences of poses. The motif level guides sets of consecutive poses to form a movement that belongs to a specific distribution using a novel motion perceptual-loss. And the choreography level selects the order of the performed movements and drives the system to follow the global structure of a dance genre. Our results demonstrate the effectiveness of our music-driven framework to generate natural and consistent movements on various dance types, having control over the content of the synthesized motions, and respecting the overall structure of the dance.

Figures (16)

• Figure 1: Music-driven motion synthesis. Our framework enables generation of rich and realistic human motions synchronized with music that form a global structure respecting the culture of a specific dance genre. Our framework is composed of three levels (bottom to top): (i) the pose level - that produces temporally coherent poses synchronized with beats, (ii) the motif level - that ensures the short movements belong to specific motion cluster (motifs) (iii) and the choreography level - that takes care of the global, long-term, structure of the dance, by selecting the next target motif to match the signature of the dance.
• Figure 2: Motion words and motifs. A whole motion sequence (displayed from bottom left to top right) is divided into short temporal segments, called motion words (here represented by one pose). Each word is clustered in the embedding space and associated with a motion motif, which is the the centroid of the cluster (here represented by the color of the word and ID number).
• Figure 3: Motion Signatures: the bars' color indicate the cluster id and the height shows the frequency of the motion word in the motion sequence. We only illustrate the distribution of the 20 most frequent words. The right column shows motion signatures of a salsa leader and left column of a salsa follower. The three bottom rows are from specific dances and the top row is the template motion signature we built for this genre.
• Figure 4: Our framework consists of three main components: pose, motif and choreography, where each is responsible for a different resolution in the motion synthesis process. The pose level generates a sequence of temporal coherent poses; the motif level ensures that a sequence of poses creates a short movement that follows a specific motion motif; the choreography level takes care of the global, long-term, structure of the dance by generating the sequence of motifs.
• Figure 5: A Gallery of selected frames extracted from our results for various dance genres. Top to bottom: salsa leader and salsa follower, Greek folk dance, modern dance, and hip-hop (from the AIST++ database).