GENMO: A GENeralist Model for Human MOtion

TL;DR

GENMO addresses the fragmentation between motion estimation and generation by unifying them in a diffusion-based generalist model. It reframes estimation as constrained generation and uses dual-mode training with estimation-guided objectives to leverage in-the-wild 2D data, enabling variable-length, multi-modal conditioning. The approach yields state-of-the-art results on global/local motion estimation and multiple generation tasks (music-to-dance, text-to-motion, in-betweening) and demonstrates bidirectional benefits where generative priors aid estimation and motion data improves generation. This framework reduces the need for 3D ground truth data and supports flexible control, with potential impact on animation, AR/VR, and video analysis.

Abstract

Human motion modeling traditionally separates motion generation and estimation into distinct tasks with specialized models. Motion generation models focus on creating diverse, realistic motions from inputs like text, audio, or keyframes, while motion estimation models aim to reconstruct accurate motion trajectories from observations like videos. Despite sharing underlying representations of temporal dynamics and kinematics, this separation limits knowledge transfer between tasks and requires maintaining separate models. We present GENMO, a unified Generalist Model for Human Motion that bridges motion estimation and generation in a single framework. Our key insight is to reformulate motion estimation as constrained motion generation, where the output motion must precisely satisfy observed conditioning signals. Leveraging the synergy between regression and diffusion, GENMO achieves accurate global motion estimation while enabling diverse motion generation. We also introduce an estimation-guided training objective that exploits in-the-wild videos with 2D annotations and text descriptions to enhance generative diversity. Furthermore, our novel architecture handles variable-length motions and mixed multimodal conditions (text, audio, video) at different time intervals, offering flexible control. This unified approach creates synergistic benefits: generative priors improve estimated motions under challenging conditions like occlusions, while diverse video data enhances generation capabilities. Extensive experiments demonstrate GENMO's effectiveness as a generalist framework that successfully handles multiple human motion tasks within a single model.

Figures (3)

• Figure 1: GENMO Model Design supports the generation of variable-length motion sequences in a single pass and enables seamless integration of multimodal conditioning signals, supporting both human motion generation and estimation.
• Figure 2: Multi-text attention enables flexible conditioning with multiple text inputs, each constrained to its specified time window.
• Figure 3: Variance of video/text conditioned predictions. Left: Intermediate predictions across 50 DDIM denoising steps. Right: Predictions with 10 different initial noises (including zero-noise). Motions are transparent except the first-step and zero-noise predictions. Video conditioning yields more deterministic outputs compared to text conditioning.