Invisible Strings: Revealing Latent Dancer-to-Dancer Interactions with Graph Neural Networks

TL;DR

The paper tackles the challenge of uncovering latent inter-dancer coupling in partnered performances by converting video data into 3D pose graphs and applying a Neural Relational Inference framework enhanced with Graph Convolutional Networks and a GRNN-based decoder, trained via self-supervised sequence reconstruction. It validates the approach on both synthetic n-body simulations and real 3D duet data, showing that inferred inter-dancer edges and hubs align with choreographic intuition and reveal patterns of tension and release. This work provides a graph-based lens for analyzing duet dynamics, with potential applications in co-creative studios, interactive performances, and dance education. Overall, it demonstrates that latent, edge-based representations can illuminate subtle partnering influences beyond explicit contact or choreography, offering new tools for designers and dancers to understand and shape collaborative movement.

Abstract

Dancing in a duet often requires a heightened attunement to one's partner: their orientation in space, their momentum, and the forces they exert on you. Dance artists who work in partnered settings might have a strong embodied understanding in the moment of how their movements relate to their partner's, but typical documentation of dance fails to capture these varied and subtle relationships. Working closely with dance artists interested in deepening their understanding of partnering, we leverage Graph Neural Networks (GNNs) to highlight and interpret the intricate connections shared by two dancers. Using a video-to-3D-pose extraction pipeline, we extract 3D movements from curated videos of contemporary dance duets, apply a dedicated pre-processing to improve the reconstruction, and train a GNN to predict weighted connections between the dancers. By visualizing and interpreting the predicted relationships between the two movers, we demonstrate the potential for graph-based methods to construct alternate models of the collaborative dynamics of duets. Finally, we offer some example strategies for how to use these insights to inform a generative and co-creative studio practice.

Figures (7)

• Figure 1: A raw 2D pose sequence from the Halpe pretrained model (26 keypoints) that shows vertex noise and even a missing dancer at one timestep.
• Figure 2: Comparison of 3D pose extractions: HybrIK (bottom) outperforms VIBE (top) in both simple (stationary) or complex (dynamic) movements.
• Figure 3: Comparison of 3D pose extractions for a dancer: minimal processing (top) vs. full pipeline (bottom), with noticeably smoother and more realistic movements.
• Figure 4: Schematic of the final model architecture, including the GCN nodes and the GRNN adapatation, inspired by the one found in the original NRI paper kipf2018neural (Figure 3, page 3).
• Figure 5: On top, original simulated trajectories and original sampled edges. On bottom, reconstruction and edge prediction results. The model accurately captured the movement and location of three particles (green, black, blue), approximated movement shape for one (orange) despite location inaccuracies, and positioned the last (red) reasonably well but without movement.