Hyperbolic Space Learning Method Leveraging Temporal Motion Priors for Human Mesh Recovery

TL;DR

This work tackles the challenge of recovering accurate 3D human meshes from video by modeling the mesh's inherent hierarchical structure in hyperbolic space. It introduces a temporal motion prior extraction module to capture both pose dynamics and image-based motion, and a hyperbolic space optimization strategy with two modules, $HPO$ and $HMO$, guided by the temporal prior. By leveraging the Poincaré Ball geometry, Möbius operations, and hyperbolic attention, the method achieves more accurate and smoother meshes, aided by a dedicated hyperbolic mesh loss. Experiments on standard benchmarks show state-of-the-art accuracy with strong generalization, particularly under challenging illumination and fast-motion conditions, demonstrating the practical impact of hyperbolic geometry for video-based human mesh recovery.

Abstract

3D human meshes show a natural hierarchical structure (like torso-limbs-fingers). But existing video-based 3D human mesh recovery methods usually learn mesh features in Euclidean space. It's hard to catch this hierarchical structure accurately. So wrong human meshes are reconstructed. To solve this problem, we propose a hyperbolic space learning method leveraging temporal motion prior for recovering 3D human meshes from videos. First, we design a temporal motion prior extraction module. This module extracts the temporal motion features from the input 3D pose sequences and image feature sequences respectively. Then it combines them into the temporal motion prior. In this way, it can strengthen the ability to express features in the temporal motion dimension. Since data representation in non-Euclidean space has been proved to effectively capture hierarchical relationships in real-world datasets (especially in hyperbolic space), we further design a hyperbolic space optimization learning strategy. This strategy uses the temporal motion prior information to assist learning, and uses 3D pose and pose motion information respectively in the hyperbolic space to optimize and learn the mesh features. Then, we combine the optimized results to get an accurate and smooth human mesh. Besides, to make the optimization learning process of human meshes in hyperbolic space stable and effective, we propose a hyperbolic mesh optimization loss. Extensive experimental results on large publicly available datasets indicate superiority in comparison with most state-of-the-art.

Figures (5)

• Figure 1: In extreme illumination scenes, our method can recover reasonable human mesh (no limb atrophy or malposition) compared to the SOTA method.
• Figure 2: The overall architecture of the our method. We use the pre-trained ResNet-50 and 3D pose estimator to extract $F$ and $P^{3d}$ of the input video $V$. Then, we extract the $TF_{motion}$ and the $P_{motion}^{3d}$ from the $F$ and the $P^{3d}$ respectively, and fuse them into the $TM_{pr}$. We apply expmap to map the input information to the hyperbolic space. With the assistance of the $TM_{pr}$ information, we make the $M_{init}$ interact and learn with $P^{3d}$ and $P_{motion}^{3d}$ respectively to optimize the mesh features. The optimized result $M_{opt}$ is mapped back to the Euclidean space and upsampled to get the accurate human mesh $M_{out}$.
• Figure 3: In outdoor fast-motion scenes, our method can recover accurate human mesh compared to the SOTA method.
• Figure 4: Visual ablation experiments on whether to extract temproal motion priors.
• Figure 5: Visual ablation experiments on whether to use Hymesh Loss.