Superman: Unifying Skeleton and Vision for Human Motion Perception and Generation

TL;DR

Superman unifies perception and generation of human motion by grounding a discrete cross-modal motion vocabulary in a Vision-Guided Motion Tokenizer (VGMT) that fuses visual evidence with 3D skeleton geometry. A single decoder-only Multi-modal LLM (MLLM) learns to handle three tasks—3D Pose Estimation from video, Motion Prediction, and Motion In-betweening—by autoregressively modeling encoded motion tokens, optionally enhanced by the Motion-Aware Fine-Tuning (MAFT) module. Empirical results on Human3.6M and zero-shot generalization to 3DPW show state-of-the-art or competitive performance across all tasks, with ablations confirming the value of visual grounding, larger model/codebook capacity, and unified multi-task training. The approach demonstrates efficient, scalable, and robust generative motion analysis that tightly links visual input to skeleton-based motion language, enabling broader applicability in motion understanding and synthesis.

Abstract

Human motion analysis tasks, such as temporal 3D pose estimation, motion prediction, and motion in-betweening, play an essential role in computer vision. However, current paradigms suffer from severe fragmentation. First, the field is split between ``perception'' models that understand motion from video but only output text, and ``generation'' models that cannot perceive from raw visual input. Second, generative MLLMs are often limited to single-frame, static poses using dense, parametric SMPL models, failing to handle temporal motion. Third, existing motion vocabularies are built from skeleton data alone, severing the link to the visual domain. To address these challenges, we introduce Superman, a unified framework that bridges visual perception with temporal, skeleton-based motion generation. Our solution is twofold. First, to overcome the modality disconnect, we propose a Vision-Guided Motion Tokenizer. Leveraging the natural geometric alignment between 3D skeletons and visual data, this module pioneers robust joint learning from both modalities, creating a unified, cross-modal motion vocabulary. Second, grounded in this motion language, a single, unified MLLM architecture is trained to handle all tasks. This module flexibly processes diverse, temporal inputs, unifying 3D skeleton pose estimation from video (perception) with skeleton-based motion prediction and in-betweening (generation). Extensive experiments on standard benchmarks, including Human3.6M, demonstrate that our unified method achieves state-of-the-art or competitive performance across all motion tasks. This showcases a more efficient and scalable path for generative motion analysis using skeletons.

Figures (21)

• Figure 1: Architecture of our Vision-Guided Motion Tokenizer (VGMT). VGMT creates a discrete motion vocabulary by jointly fusing information from two modalities. A Skeleton Encoder ($E_s$) captures geometry while a Visual-Skeleton Attention (VSA) module and a subsequent Visual Encoder ($E_v$) ground the pose in visual features. The fused representation is quantized against a learnable hybrid codebook, and a decoder reconstructs the 3D poses.
• Figure 2: Network architecture and training paradigm. Superman fine-tune a single LLM to integrate information from text, video, and 3D skeleton modalities. Optionally, a Motion-Aware Fine-Tuning (MAFT) module can be integrated. With $<$0.2% extra parameters, MAFT enhances motion perception by enabling cross-video-motion fusion, leading to substantial improvement on tasks with visual input, as validated by the experiment results.
• Figure 3: Two standard pipelines for 3D pose estimation. "2D PE" means 2D pose estimator. "skel" is short for skeleton.
• Figure 4: Qualitative results for pose estimation on Human3.6M. Our method with MAFT is compared with HiC liu2025human, the current SoTA. Both methods use video and 2D poses as inputs and leverage a fixed 2D pose estimator.
• Figure 5: Qualitative results for generalizing to motion prediction on 3DPW (unseen dataset). Our method is compared with SiC liu2025human, the current SoTA on the task.