Two-in-One: Unified Multi-Person Interactive Motion Generation by Latent Diffusion Transformer
Boyuan Li, Xihua Wang, Ruihua Song, Wenbing Huang
TL;DR
This study tackles multi-person interactive motion generation by moving from multi-branch, interaction-bridged pipelines to a unified latent-space approach. It introduces an Interaction Variational Autoencoder (InterVAE) to compress two-person motions into a shared latent representation $z \in \mathbb{R}^{n\times d}$ of about $\approx\frac{1}{10}$ the input size, and a Conditional Interaction Latent Diffusion Model (InterLDM) based on a Latent Diffusion Transformer to generate $z$ conditioned on text, optimized with $L_{\text{VAE}}$ and $L_{\text{LDM}}$ losses. Text conditioning is encoded via frozen CLIP-ViT-L-14 and T5-small encoders, and generation uses a 25-step diffusion with DPMSolver++ and classifier-free guidance. On the InterHumanintergen dataset, InterLDM achieves superior R Precision, FID, MM Dist, and Diversity compared with baselines, while delivering over 4x faster inference than the best two-branch method. The approach offers a scalable, high-quality, text-conditioned solution for realistic multi-person animation with potential impact on games, film, and VR workflows.
Abstract
Multi-person interactive motion generation, a critical yet under-explored domain in computer character animation, poses significant challenges such as intricate modeling of inter-human interactions beyond individual motions and generating two motions with huge differences from one text condition. Current research often employs separate module branches for individual motions, leading to a loss of interaction information and increased computational demands. To address these challenges, we propose a novel, unified approach that models multi-person motions and their interactions within a single latent space. Our approach streamlines the process by treating interactive motions as an integrated data point, utilizing a Variational AutoEncoder (VAE) for compression into a unified latent space, and performing a diffusion process within this space, guided by the natural language conditions. Experimental results demonstrate our method's superiority over existing approaches in generation quality, performing text condition in particular when motions have significant asymmetry, and accelerating the generation efficiency while preserving high quality.