OmniMotion-X: Versatile Multimodal Whole-Body Motion Generation

TL;DR

OmniMotion-X addresses multimodal whole-body motion generation by unifying text, audio, trajectories, and a motion prior within a single autoregressive diffusion Transformer. It is empowered by OmniMoCap-X, a large-scale, high-quality multimodal mocap dataset of $64.3$M frames and $286.2$ hours across $28$ sources, standardized to SMPL-X at $30$ FPS with hierarchical GPT-4o captions. A weak-to-strong progressive conditioning strategy enables learning across multi-granularity inputs, achieving state-of-the-art results on T2M, S2G, M2D, and GSTC, and enabling interactive long-duration generation. Limitations include lack of scene/object interaction constraints and slower inference, pointing to future work in integrating scene constraints and improving efficiency.

Abstract

This paper introduces OmniMotion-X, a versatile multimodal framework for whole-body human motion generation, leveraging an autoregressive diffusion transformer in a unified sequence-to-sequence manner. OmniMotion-X efficiently supports diverse multimodal tasks, including text-to-motion, music-to-dance, speech-to-gesture, and global spatial-temporal control scenarios (e.g., motion prediction, in-betweening, completion, and joint/trajectory-guided synthesis), as well as flexible combinations of these tasks. Specifically, we propose the use of reference motion as a novel conditioning signal, substantially enhancing the consistency of generated content, style, and temporal dynamics crucial for realistic animations. To handle multimodal conflicts, we introduce a progressive weak-to-strong mixed-condition training strategy. To enable high-quality multimodal training, we construct OmniMoCap-X, the largest unified multimodal motion dataset to date, integrating 28 publicly available MoCap sources across 10 distinct tasks, standardized to the SMPL-X format at 30 fps. To ensure detailed and consistent annotations, we render sequences into videos and use GPT-4o to automatically generate structured and hierarchical captions, capturing both low-level actions and high-level semantics. Extensive experimental evaluations confirm that OmniMotion-X significantly surpasses existing methods, demonstrating state-of-the-art performance across multiple multimodal tasks and enabling the interactive generation of realistic, coherent, and controllable long-duration motions.

Figures (5)

• Figure 1: Overview of OmniMotion-X, a unified multimodal autoregressive transformer diffusion model for whole-body human motion generation. OmniMotion-X integrates text, global motion, speech, music, and reference motion as conditions through condition-specific encoders mapped into a unified space. The model fuses multimodal information to produce coherent motion, with spatial-temporal guidance ensuring consistent global motion characteristics.
• Figure 2: We propose the weak-to-strong progressive training strategy, establishing motion-semantic alignment with text, followed by progressive integration of stronger multimodal signals (reference motion, global motion, speech, music) for enhanced generation quality and controllability.
• Figure 3: Diverse motion synthesis capabilities of OmniMotion-X.OmniMotion-X supports multiple tasks: (a) text-to-motion, (b) speech-to-gesture, (c) music-to-dance, (d) trajectory-guided motion, (e) motion in-betweening, and (f) motion prediction.
• Figure 4: Example of GPT-4o Captioning Quality.
• Figure 5: From left to right, HHI, HOI, and HSI of our dataset.