A Unified Framework for Motion Reasoning and Generation in Human Interaction

TL;DR

VIM introduces a unified motion-language framework that processes and generates two-person interactive motions within multi-turn conversations. A new dataset, Inter-MT2, provides 82K multi-turn motion-text instances (153K motions) to train and evaluate the model across motion reasoning, editing, and traditional tasks. The approach hinges on a three-stage pipeline—motion tokenizer training, cross-modal alignment, and instruction-tuning—coupled with a diffusion-based motion generator and an LLM backbone. Empirical results show VIM outperforms two-stage baselines on reasoning, editing, and M2T/T2M tasks, and can extend to multi-human motion generation, suggesting broad applicability in robotics, VR, and human-computer interaction. The work includes extensive ablations, data collection pipelines, and implementation details, with public release of data and code for future research.

Abstract

Recent advancements in large language models (LLMs) have significantly improved their ability to generate natural and contextually relevant text, enabling more human-like AI interactions. However, generating and understanding interactive human-like motion, where multiple individuals engage in coordinated movements, remains challenging due to the complexity of modeling these interactions. Additionally, a unified and versatile model is needed to handle diverse interactive scenarios, such as chat systems that dynamically adapt to user instructions and assigned roles. To address these challenges, we introduce VIM, the Versatile Interactive Motion-language model, which integrates both language and motion modalities to effectively understand, generate, and control interactive motions in multi-turn conversational contexts. Unlike previous studies that primarily focus on uni-directional tasks such as text-to-motion or motion-to-text, VIM employs a unified architecture capable of simultaneously understanding and generating both motion and text modalities. Given the absence of an appropriate dataset to support this task, we introduce Inter-MT2, a large-scale instruction-tuning dataset containing 82.7K multi-turn interactive motion instructions, covering 153K interactive motion samples. Inter-MT2 spans diverse instructional scenarios, including motion editing, question answering, and story generation, leveraging off-the-shelf large language models and motion diffusion models to construct a broad set of interactive motion instructions. We extensively evaluate the versatility of VIM across multiple interactive motion-related tasks, including motion-to-text, text-to-motion, reaction generation, motion editing, and reasoning about motion sequences.

Figures (22)

• Figure 1: We introduce VIM, the Versatile Interactive Motion-language model, a unified architecture that combines language and motion for two-person interactive scenarios. The figure highlights its capabilities across various tasks including motion-to-text, text-to-motion, reaction generation, motion editing, and multi-turn motion reasoning, all within a single framework.
• Figure 2: Statistics and data sample from $\text{I}$nter-$\text{MT}^2$.
• Figure 3: An overview of VIM, illustrating its versatile capability to flexibly process and generate interactive motions and texts in an auto-regressive manner. We omit the motion tokenizer, which converts raw motion sequences into discrete motion tokens, for clarity. VIM covers versatile motion tasks involving both motion and textual modalities across multiple conversational turns.
• Figure 4: Tokenization of interactive motions.
• Figure 5: Generated samples for interactive motion reasoning task. This example shows how VIM explains behaviors and their motivations, demonstrating a deeper understanding of scenarios by incorporating context from prior interactions.