Towards Geometry-Aware and Motion-Guided Video Human Mesh Recovery

TL;DR

HMRMamba tackles the instability of intermediate 3D pose anchors and weak temporal modeling in video-based 3D Human Mesh Recovery. It introduces a two-stage framework that combines a Geometry-Aware Lifting Module with STA-Mamba to produce geometrically grounded 3D pose anchors, and a Motion-guided Reconstruction Network to exploit full temporal dynamics via motion-aware attention. Key contributions include the Dual-Scan Mamba blocks that merge global context with kinematic priors, explicit/implicit motion representations for attention, and strong state-of-the-art results on 3DPW, MPI-INF-3DHP, and Human3.6M with notable efficiency. The approach enhances robustness under occlusion and motion blur and offers practical benefits for real-time or large-scale deployment.

Abstract

Existing video-based 3D Human Mesh Recovery (HMR) methods often produce physically implausible results, stemming from their reliance on flawed intermediate 3D pose anchors and their inability to effectively model complex spatiotemporal dynamics. To overcome these deep-rooted architectural problems, we introduce HMRMamba, a new paradigm for HMR that pioneers the use of Structured State Space Models (SSMs) for their efficiency and long-range modeling prowess. Our framework is distinguished by two core contributions. First, the Geometry-Aware Lifting Module, featuring a novel dual-scan Mamba architecture, creates a robust foundation for reconstruction. It directly grounds the 2D-to-3D pose lifting process with geometric cues from image features, producing a highly reliable 3D pose sequence that serves as a stable anchor. Second, the Motion-guided Reconstruction Network leverages this anchor to explicitly process kinematic patterns over time. By injecting this crucial temporal awareness, it significantly enhances the final mesh's coherence and robustness, particularly under occlusion and motion blur. Comprehensive evaluations on 3DPW, MPI-INF-3DHP, and Human3.6M benchmarks confirm that HMRMamba sets a new state-of-the-art, outperforming existing methods in both reconstruction accuracy and temporal consistency while offering superior computational efficiency.

Figures (4)

• Figure 1: Conceptual comparison of HMR pipelines. (a) Existing methods often suffer from inaccurate intermediate geometry and poor occlusion handling, leading to flawed reconstructions with wrong body sizes and unnatural poses. (b) Our HMRMamba leverages temporal evolution for accurate geometry perception and motion modeling, resulting in robust and reasonable mesh recovery.
• Figure 2: The two-stage architecture of HMRMamba. (1) Geometry-Aware Lifting Module (top row): To address the problem of unreliable anchors, our novel STA-Mamba infuses geometric cues from image features into the lifting process. This produces a robust, geometrically-grounded 3D pose sequence that serves as a stable anchor. (2) Motion-guided Reconstruction Network (bottom row): Leveraging this stable anchor, a motion-aware attention mechanism models kinematic dynamics. This injects temporal awareness into the final regression to ensure a coherent and physically plausible mesh.
• Figure 3: Detailed architecture of our Dual-Scan Mamba Block. The Global Scan branch processes the sequence linearly to capture holistic, long-range dependencies. In contrast, the Local Scan branch employs a non-sequential scanning order that follows the human kinematic tree, as illustrated on the right. This novel local scan explicitly models anatomical constraints by traversing joints along natural limbs. The outputs of these two complementary branches are fused via element-wise multiplication and addition, producing a global context and local structural plausibility representation.
• Figure 4: Qualitative results on Human3.6m dataset. HMR under occlusion from two viewpoints.