Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Masked Modeling for Human Motion Recovery Under Occlusions

Zhiyin Qian, Siwei Zhang, Bharat Lal Bhatnagar, Federica Bogo, Siyu Tang

TL;DR

This work tackles occlusion-robust monocular human motion reconstruction by reframing the task as video-conditioned generative modeling. It introduces MoRo, a masked generative transformer architecture that fuses a trajectory-aware motion prior, an image-conditioned pose prior, and a video-conditioned cross-modal decoder to recover SMPL-X pose tokens and a global trajectory from RGB videos. The approach is trained in stages across MoCap, image-pose, and video datasets, and employs iterative inference to handle occlusions efficiently, achieving real-time performance at 70 FPS on a single H200 GPU while outperforming state-of-the-art baselines on occluded sequences and remaining competitive in unobstructed scenarios. The results demonstrate strong occlusion handling with improved motion realism and global-consistency, offering a practical solution for AR/VR, robotics, and digital content creation under real-world occlusions, albeit with current limitations to static camera setups.

Abstract

Human motion reconstruction from monocular videos is a fundamental challenge in computer vision, with broad applications in AR/VR, robotics, and digital content creation, but remains challenging under frequent occlusions in real-world settings. Existing regression-based methods are efficient but fragile to missing observations, while optimization- and diffusion-based approaches improve robustness at the cost of slow inference speed and heavy preprocessing steps. To address these limitations, we leverage recent advances in generative masked modeling and present MoRo: Masked Modeling for human motion Recovery under Occlusions. MoRo is an occlusion-robust, end-to-end generative framework that formulates motion reconstruction as a video-conditioned task, and efficiently recover human motion in a consistent global coordinate system from RGB videos. By masked modeling, MoRo naturally handles occlusions while enabling efficient, end-to-end inference. To overcome the scarcity of paired video-motion data, we design a cross-modality learning scheme that learns multi-modal priors from a set of heterogeneous datasets: (i) a trajectory-aware motion prior trained on MoCap datasets, (ii) an image-conditioned pose prior trained on image-pose datasets, capturing diverse per-frame poses, and (iii) a video-conditioned masked transformer that fuses motion and pose priors, finetuned on video-motion datasets to integrate visual cues with motion dynamics for robust inference. Extensive experiments on EgoBody and RICH demonstrate that MoRo substantially outperforms state-of-the-art methods in accuracy and motion realism under occlusions, while performing on-par in non-occluded scenarios. MoRo achieves real-time inference at 70 FPS on a single H200 GPU.

Masked Modeling for Human Motion Recovery Under Occlusions

TL;DR

This work tackles occlusion-robust monocular human motion reconstruction by reframing the task as video-conditioned generative modeling. It introduces MoRo, a masked generative transformer architecture that fuses a trajectory-aware motion prior, an image-conditioned pose prior, and a video-conditioned cross-modal decoder to recover SMPL-X pose tokens and a global trajectory from RGB videos. The approach is trained in stages across MoCap, image-pose, and video datasets, and employs iterative inference to handle occlusions efficiently, achieving real-time performance at 70 FPS on a single H200 GPU while outperforming state-of-the-art baselines on occluded sequences and remaining competitive in unobstructed scenarios. The results demonstrate strong occlusion handling with improved motion realism and global-consistency, offering a practical solution for AR/VR, robotics, and digital content creation under real-world occlusions, albeit with current limitations to static camera setups.

Abstract

Human motion reconstruction from monocular videos is a fundamental challenge in computer vision, with broad applications in AR/VR, robotics, and digital content creation, but remains challenging under frequent occlusions in real-world settings. Existing regression-based methods are efficient but fragile to missing observations, while optimization- and diffusion-based approaches improve robustness at the cost of slow inference speed and heavy preprocessing steps. To address these limitations, we leverage recent advances in generative masked modeling and present MoRo: Masked Modeling for human motion Recovery under Occlusions. MoRo is an occlusion-robust, end-to-end generative framework that formulates motion reconstruction as a video-conditioned task, and efficiently recover human motion in a consistent global coordinate system from RGB videos. By masked modeling, MoRo naturally handles occlusions while enabling efficient, end-to-end inference. To overcome the scarcity of paired video-motion data, we design a cross-modality learning scheme that learns multi-modal priors from a set of heterogeneous datasets: (i) a trajectory-aware motion prior trained on MoCap datasets, (ii) an image-conditioned pose prior trained on image-pose datasets, capturing diverse per-frame poses, and (iii) a video-conditioned masked transformer that fuses motion and pose priors, finetuned on video-motion datasets to integrate visual cues with motion dynamics for robust inference. Extensive experiments on EgoBody and RICH demonstrate that MoRo substantially outperforms state-of-the-art methods in accuracy and motion realism under occlusions, while performing on-par in non-occluded scenarios. MoRo achieves real-time inference at 70 FPS on a single H200 GPU.
Paper Structure (17 sections, 5 equations, 2 figures, 3 tables)

This paper contains 17 sections, 5 equations, 2 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Motion Representation
  4. Method
  5. Per-frame Image Conditioning
  6. Trajectory-aware Motion Prior
  7. Video-conditioned Masked Transformer
  8. Inference
  9. Multi-stage Training
  10. Experiments
  11. Datasets
  12. Implementation Details
  13. Evaluation Metrics
  14. Baselines
  15. Results
  16. ...and 2 more sections

Figures (2)

  • Figure 1: Overview of our masked transformer, which consists of three main components: the image encoder, the motion encoder and the decoder. Given a monocular video sequence, we utilize the image encoder to extract per-frame image features and estimate a coarse global trajectory, which is canonicalized and serves as the input to the motion encoder (Sec. \ref{['sec:image-encoder']}). Along with masked local pose tokens, the motion encoder encodes a trajectory-aware motion prior via recovering the complete local pose tokens and denoising the global trajectory (Sec. \ref{['sec:motion-encoder']}). The cross-modality decoder fuses the intermediate feature from both encoders via a spatial-temporal transformer to refine the camera-space global trajectory and predict a conditional categorical distribution for sampling the local pose tokens, which are then smoothed for enhanced motion realism (Sec. \ref{['sec:full-model']}).
  • Figure 2: Qualitative examples on EgoBody (row 1, 2) and RICH (row 3). For row 1-2, from left to right corresponds to WHAM-Cam, GVHMR-Cam, PromptHMR and ours. For row 3, from left to right corresponds to WHAM-Cam, WHAM-World, GVHMR-Cam, GVHMR-World, PromptHMR and ours.