Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Dyads: Artist-Centric, AI-Generated Dance Duets

Zixuan Wang, Luis Zerkowski, Ilya Vidrin, Mariel Pettee

TL;DR

Dyads tackles the challenge of modeling choreographic duets by integrating a probability-based Variational Autoencoder with Transformer decoders to generate a partner conditioned on an input sequence. The architecture uses three VAEs (two for individuals and one for their interaction) and a velocity loss to improve motion coherence, with training guided by a probabilistic schedule and an autoregressive generation process. Pose data are extracted from four duet videos using AlphaPose and HybrIK, then processed into robust 3D joint sequences; the model is trained on these sequences and released as open-source to support interdisciplinary collaboration. Quantitative results show increasing MSE over longer horizons, while qualitative outputs reveal diverse, partner-responsive movements, albeit with occasional physically implausible artifacts—highlighting both the potential and current limits of AI-assisted duets. The work foregrounds artist involvement, ethical considerations, and a co-creative workflow, offering a practical framework for integrating artistic goals with AI research in dance.

Abstract

Existing AI-generated dance methods primarily train on motion capture data from solo dance performances, but a critical feature of dance in nearly any genre is the interaction of two or more bodies in space. Moreover, many works at the intersection of AI and dance fail to incorporate the ideas and needs of the artists themselves into their development process, yielding models that produce far more useful insights for the AI community than for the dance community. This work addresses both needs of the field by proposing an AI method to model the complex interactions between pairs of dancers and detailing how the technical methodology can be shaped by ongoing co-creation with the artistic stakeholders who curated the movement data. Our model is a probability-and-attention-based Variational Autoencoder that generates a choreographic partner conditioned on an input dance sequence. We construct a custom loss function to enhance the smoothness and coherence of the generated choreography. Our code is open-source, and we also document strategies for other interdisciplinary research teams to facilitate collaboration and strong communication between artists and technologists.

Dyads: Artist-Centric, AI-Generated Dance Duets

TL;DR

Dyads tackles the challenge of modeling choreographic duets by integrating a probability-based Variational Autoencoder with Transformer decoders to generate a partner conditioned on an input sequence. The architecture uses three VAEs (two for individuals and one for their interaction) and a velocity loss to improve motion coherence, with training guided by a probabilistic schedule and an autoregressive generation process. Pose data are extracted from four duet videos using AlphaPose and HybrIK, then processed into robust 3D joint sequences; the model is trained on these sequences and released as open-source to support interdisciplinary collaboration. Quantitative results show increasing MSE over longer horizons, while qualitative outputs reveal diverse, partner-responsive movements, albeit with occasional physically implausible artifacts—highlighting both the potential and current limits of AI-assisted duets. The work foregrounds artist involvement, ethical considerations, and a co-creative workflow, offering a practical framework for integrating artistic goals with AI research in dance.

Abstract

Existing AI-generated dance methods primarily train on motion capture data from solo dance performances, but a critical feature of dance in nearly any genre is the interaction of two or more bodies in space. Moreover, many works at the intersection of AI and dance fail to incorporate the ideas and needs of the artists themselves into their development process, yielding models that produce far more useful insights for the AI community than for the dance community. This work addresses both needs of the field by proposing an AI method to model the complex interactions between pairs of dancers and detailing how the technical methodology can be shaped by ongoing co-creation with the artistic stakeholders who curated the movement data. Our model is a probability-and-attention-based Variational Autoencoder that generates a choreographic partner conditioned on an input dance sequence. We construct a custom loss function to enhance the smoothness and coherence of the generated choreography. Our code is open-source, and we also document strategies for other interdisciplinary research teams to facilitate collaboration and strong communication between artists and technologists.

Paper Structure

This paper contains 17 sections, 5 equations, 4 figures, 1 table.

Table of Contents

  1. Introduction
  2. Related Work
  3. Methods
  4. Pose Extraction
  5. Model Architecture
  6. Training and Inference
  7. Variational Autoencoder Configuration
  8. Transformer Decoder
  9. Losses
  10. Implementation Details
  11. Results
  12. Quantitative results
  13. Qualitative results
  14. Discussion
  15. Artist Involvement
  16. ...and 2 more sections

Figures (4)

  • Figure 1: Extraction results from AlphaPose showing keypoints and body mesh.
  • Figure 2: Movement sequence refinement illustration. The top row shows the original movement data, while the bottom row shows the refined results after processing.
  • Figure 3: Network structure.
  • Figure 4: Examples of several generated test sequences from the model in which the figure in blue is a generated partner conditioned on the movements of the figure in black. The movements range in physical plausibility, ranging from slow and deliberate movements to more complex and even, at times, unphysical dynamics.