Exploring AI-assisted Ideation and Prototyping for Choreography

TL;DR

Choreography is a multimodal creative process constrained by time and resources, demanding both cognitive ideation and physical prototyping. The authors present an AI-assisted choreography-support system that generates diverse dance sequences from text or video and enables iterative digital prototyping through a web-based UI, with 2D videos and 3D animated meshes as outputs. The backend combines a Motion Diffusion Model fine-tuned on relevant datasets and analyzes inputs with CLIP and VIBE, enabling editing operations such as style transfer, partial-body edits, and sequence blending. An initial evaluation with six professional choreographers highlights benefits for ideation speed and reproducibility while revealing limitations in conveying emotion digitally and in balancing AI-generated content with rapid search. The work demonstrates a pathway toward low-physical-load, rapid-prototype choreography tooling and identifies directions for richer emotion rendering and extended evaluative studies.

Abstract

Choreography creation is a multimodal endeavor, demanding cognitive abilities to develop creative ideas and technical expertise to convert choreographic ideas into physical dance movements. Previous endeavors have sought to reduce the complexities in the choreography creation process in both dimensions. Among them, non-AI-based systems have focused on reinforcing cognitive activities by helping analyze and understand dance movements and augmenting physical capabilities by enhancing body expressivity. On the other hand, AI-based methods have helped the creation of novel choreographic materials with generative AI algorithms. The choreography creation process is constrained by time and requires a rich set of resources to stimulate novel ideas, but the need for iterative prototyping and reduced physical dependence have not been adequately addressed by prior research. Recognizing these challenges and the research gap, we present an innovative AI-based choreography-support system. Our goal is to facilitate rapid ideation by utilizing a generative AI model that can produce diverse and novel dance sequences. The system is designed to support iterative digital dance prototyping through an interactive web-based user interface that enables the editing and modification of generated motion. We evaluated our system by inviting six choreographers to analyze its limitations and benefits and present the evaluation results along with potential directions for future work.

Figures (3)

• Figure 1: User interface. The interface accepts text and video as input. Typing text descriptions leads to dance generation, and uploading videos offers converted 3D dance sequences for editing. After dance sequences are generated or uploaded, users can further edit them using the editing options. Dance extension is allowed by typing in the length to extend by up to 5 seconds. For style control, users can pick styles from a drop-down menu. To edit partial body movements, users can choose a body part and describe how they want to change it in the text box below. After creating dance sequences, users can add them to the Gallery for future use, shown as thumbnails. If users select dance sequences from the Gallery, they can blend them. On the right, users can view and interact with the generated dance sequences represented by a digital avatar. The visibility of the avatar's mesh and skeleton is adjustable via the checkboxes. Three types of avatar meshes are available in our current prototype: SMPL SMPL_2015 male, female, and Mixamo mixamo mesh. Lastly, users can download generated dance sequences as 2D videos and 3D animated meshes.
• Figure 2: Results of a dance sequence generated from "A man is dancing ballet" and edited sequences based on it. (a) The original dance sequence is 10 seconds, shown in gray, and the extended segment by 5 seconds is blue. (b) The style "angry" is applied to the original dance sequence. (c) The partial body movements are altered according to "Keep the arms raised up". (d) - (f) The blending results of the original sequence and a 10-second dance sequence generated from "A man is dancing hip-hop" in blue, with a 5-second connecting dance sequence merging them in green.
• Figure 3: Technical details. The system's user interface and backend server are connected via a TCP socket, shown as dashed arrows. The user interface sends users' text descriptions, uploaded videos, and editing operations to the backend and receives generated dance sequences for display and interaction. The backend server encodes each text description from the query queue with CLIP radford2021learning embeddings and converts dance movements from the video files into 3D sequences using VIBE kocabas2020vibe. These inputs and user-initiated edit operations are fed into the fine-tuned MDM tevet2023human to generate dance sequences. Dataflows within the backend server are represented as solid arrows.