Choreographing the Digital Canvas: A Machine Learning Approach to Artistic Performance

TL;DR

The paper tackles AI-assisted choreography by introducing an attribute-conditioned cyclic CVAE that generates phase-aware 3D motions for performances, focusing on a three-phase falling movement. It presents a unique falling-movement dataset labeled with Impact, Glitch, and Fall and represented in SMPL, plus an interactive web interface for visualization and artistic control. Quantitatively, the approach achieves higher accuracy and diversity than a baseline while exhibiting higher FID due to data limitations, and qualitative evaluations indicate improved plausibility and smoothness, especially with the concatenation fusion variant. This work enables artist-driven motion generation and visualization, expanding access to sophisticated motion design while highlighting data and augmentation challenges for future improvements.

Abstract

This paper introduces the concept of a design tool for artistic performances based on attribute descriptions. To do so, we used a specific performance of falling actions. The platform integrates a novel machine-learning (ML) model with an interactive interface to generate and visualize artistic movements. Our approach's core is a cyclic Attribute-Conditioned Variational Autoencoder (AC-VAE) model developed to address the challenge of capturing and generating realistic 3D human body motions from motion capture (MoCap) data. We created a unique dataset focused on the dynamics of falling movements, characterized by a new ontology that divides motion into three distinct phases: Impact, Glitch, and Fall. The ML model's innovation lies in its ability to learn these phases separately. It is achieved by applying comprehensive data augmentation techniques and an initial pose loss function to generate natural and plausible motion. Our web-based interface provides an intuitive platform for artists to engage with this technology, offering fine-grained control over motion attributes and interactive visualization tools, including a 360-degree view and a dynamic timeline for playback manipulation. Our research paves the way for a future where technology amplifies the creative potential of human expression, making sophisticated motion generation accessible to a wider artistic community.

Figures (10)

• Figure 1: Dynamic motion capture sequence illustrating the transition between RGB recording to MoCap to a refined 3D animation. It demonstrates our model's comprehensive process, capturing the nuanced dynamics of human movements and translating them into realistic animated sequences.
• Figure 2: Attribute-conditioned Variational AUtoencoder model architecture: Taking a sequence of body poses, we split them into inputs for different phases. The model learns to reproduce the movement specifically for each phase. The final multi-phase sequences are the concatenation of all outputs from the three phases. Improved from the ACTOR model petrovich2021action, we employed a cyclic design (indicated by the dotted red line) where the last frame from the previous phase serves as the initial guidance of the generation.
• Figure 3: Interface of the 3D human motion visualization tool showcasing the model and attribute configuration selection menu. Users can select between SMPL male and female models and customize the motion attribute for impact location, impact attribute, glitch attribute, and fall attribute. The interactive 3D viewer on the right displays the generated pose according to the selected attributes, with a slider for navigating the motion timeline.
• Figure 4: Impact Location: arms; Impact Attribute: prick; Glitch Attribute: flail; Fall Attribute: release
• Figure 5: Impact Location: torso; Impact Attribute: push; Glitch Attribute: contort; Fall Attribute: surrender