DanceGen: Supporting Choreography Ideation and Prototyping with Generative AI

TL;DR

DanceGen tackles the underexplored choreography preparation stage by combining multimodal input, editable diffusion-based motion generation, and comprehensive documentation in a web-based tool. It builds on a fine-tuned Motion Diffusion Model (MDM) and CLIP-based text embeddings to generate, edit, extend, blend, and style dance sequences, while also enabling 2D and 3D documentation and export. A formative study informs design goals, and a user study with expert choreographers demonstrates benefits in ideation, iterative prototyping, and collaboration, alongside limitations in AI fidelity and intent understanding. The work advances AI-driven choreographic ideation by enabling iterative, interactive, and shareable digital materials that can integrate with physical prototyping workflows, offering a practical pathway to accelerate early-stage choreography while highlighting areas for future improvement.

Abstract

Choreography creation requires high proficiency in artistic and technical skills. Choreographers typically go through four stages to create a dance piece: preparation, studio, performance, and reflection. This process is often individualized, complicated, and challenging due to multiple constraints at each stage. To assist choreographers, most prior work has focused on designing digital tools to support the last three stages of the choreography process, with the preparation stage being the least explored. To address this research gap, we introduce an AI-based approach to assist the preparation stage by supporting ideation, creating choreographic prototypes, and documenting creative attempts and outcomes. We address the limitations of existing AI-based motion generation methods for ideation by allowing generated sequences to be edited and modified in an interactive web interface. This capability is motivated by insights from a formative study we conducted with seven choreographers. We evaluated our system's functionality, benefits, and limitations with six expert choreographers. Results highlight the usability of our system, with users reporting increased efficiency, expanded creative possibilities, and an enhanced iterative process. We also identified areas for improvement, such as the relationship between user intent and AI outcome, intuitive and flexible user interaction design, and integration with existing physical choreography prototyping workflows. By reflecting on the evaluation results, we present three insights that aim to inform the development of future AI systems that can empower choreographers.

Figures (8)

• Figure 1: Choreography creation process, as outlined by prior research ciolfi2016choreographers, consisting of four stages: preparation, studio, performance, and reflection. These stages are interconnected and occur recursively, as indicated by the arrows.
• Figure 2: DanceGen design process. The formative study findings and our design goals are presented in Section \ref{['sec:formative_study']}. These design goals subsequently guide the development of the system's functionality. Additionally, feedback from the formative study, including participants' choreographic methods and challenges, motivates certain system features. Detailed information on the DanceGen system and its functionality is introduced in Section \ref{['sec:dancegen_system']}.
• Figure 3: DanceGen user interface and functionality. On the left, users can generate dance sequences for ideation based on text descriptions. They can further edit dance sequences using the editing options for prototyping. Specifically, dance extension is facilitated by specifying the desired extension length, up to 5 seconds. Style control is achieved through a drop-down menu for users to select different movement styles. Partial body movement editing is allowed by choosing a body part and providing corresponding descriptions in the text box below. After creating dance sequences and adding them to the Gallery, users can select dance sequences to blend them. On the right, users can view and interact with the generated dance sequences, represented by a digital avatar. The visibility of the avatar's mesh and skeleton can be adjusted using checkboxes on the far right. The current prototype offers three types of avatar meshes. Lastly, users can download the generated dance sequences in both 2D and 3D formats.
• Figure 4: Results created by user study participants. (a) and (b) Dance sequences are represented by the SMPL male mesh SMPL_2015, generated from "Raise your left heel, keeping only your toes touching the ground. Simultaneously, slide your right foot backward", and edited with the partial body condition "Maintain an upright position with your torso". (c) and (d) Dance sequences are shown by the SMPL female mesh SMPL_2015, generated from "Chest bumping" and "A person does a hitch kick". (e), (f), and (g) Dance blending outcomes of "A person performs a step ball change" and "A person is dancing hip-hop" with a connecting sequence in between, illustrated using the Mixamo mesh mixamo. Each dance sequence is depicted in five frames, while the connecting sequence is presented in three frames.
• Figure 5: Dance generation and editing results. (a) Generated dance sequence shown as five frames based on "A person performs a plié and then a backflip". (b) The original dance sequence is extended by 5 seconds, with the extended portion depicted by three frames. (c) Style control, strutting, is applied to the original dance sequence. (d) Partial body editing based on "Keep the arms raised". (e), (f), and (g) A dance sequence generated from "A person performs a kick ball change" is seamlessly blended with the original dance sequence through a 5-second connecting sequence shown as three frames.