GenHMR: Generative Human Mesh Recovery

TL;DR

This work tackles the ill-posed problem of monocular 3D human mesh recovery by reframing it as an image-conditioned generative task. It introduces GenHMR, consisting of a pose tokenizer based on VQ-VAE and an image-conditioned masked transformer that learns conditional distributions over discrete pose tokens and employs uncertainty-guided iterative decoding plus 2D pose-guided refinement. The method achieves state-of-the-art results on standard benchmarks such as Human3.6M, 3DPW, and EMDB, with substantial reductions in MPJPE and MVE and strong generalization to in-the-wild data. By explicitly modeling and reducing 2D-to-3D uncertainties, GenHMR provides robust 3D mesh reconstructions in challenging poses and occlusions, with practical implications for animation, AR/VR, and biomechanics.

Abstract

Human mesh recovery (HMR) is crucial in many computer vision applications; from health to arts and entertainment. HMR from monocular images has predominantly been addressed by deterministic methods that output a single prediction for a given 2D image. However, HMR from a single image is an ill-posed problem due to depth ambiguity and occlusions. Probabilistic methods have attempted to address this by generating and fusing multiple plausible 3D reconstructions, but their performance has often lagged behind deterministic approaches. In this paper, we introduce GenHMR, a novel generative framework that reformulates monocular HMR as an image-conditioned generative task, explicitly modeling and mitigating uncertainties in the 2D-to-3D mapping process. GenHMR comprises two key components: (1) a pose tokenizer to convert 3D human poses into a sequence of discrete tokens in a latent space, and (2) an image-conditional masked transformer to learn the probabilistic distributions of the pose tokens, conditioned on the input image prompt along with randomly masked token sequence. During inference, the model samples from the learned conditional distribution to iteratively decode high-confidence pose tokens, thereby reducing 3D reconstruction uncertainties. To further refine the reconstruction, a 2D pose-guided refinement technique is proposed to directly fine-tune the decoded pose tokens in the latent space, which forces the projected 3D body mesh to align with the 2D pose clues. Experiments on benchmark datasets demonstrate that GenHMR significantly outperforms state-of-the-art methods. Project website can be found at https://m-usamasaleem.github.io/publication/GenHMR/GenHMR.html

Figures (11)

• Figure 1: State of the art (SOTA) methods, HMR2.0 goel2023humans and TokenHMR dwivedi2024tokenhmr, leverage vision transformers to recover 3D human mesh from a single image. The errors, highlighted by red circles, reveal the limitations of these SOTA approaches in handling unusual poses or ambiguous scenarios. Our method, GenHMR, overcomes these challenges by explicitly modeling and mitigating uncertainties in the 2D-to-3D mapping process, resulting in more accurate and robust 3D pose reconstructions in such complex scenarios.
• Figure 2: GenHMR Training Phase. GenHMR consists of two key components: (1) a Pose Tokenizer that encodes 3D human poses into a sequence of discrete tokens within a latent space, and (2) an Image-Conditioned Masked Transformer that models the probabilistic distributions of these tokens, conditioned on the input image and a partially masked token sequence.
• Figure 3: Our inference strategy comprises two key stages: (1) Uncertainty-Guided Sampling, which iteratively samples high-confidence pose tokens based on their probabilistic distributions, and (2) 2D Pose-Guided Refinement, which fine-tunes the sampled pose tokens to further minimize 3D reconstruction uncertainty by ensuring consistency between the 3D body mesh and 2D pose estimates.
• Figure 4: Impact of 2D Pose-Guided Refinement on 3D pose reconstruction. Red circles highlight areas of errors after each refinement iteration, which showcases how the method progressively refines these poses. By fine-tuning pose tokens to align the 3D pose with 2D detections, our method iteratively reduces uncertainties and improves accuracy. Significant improvements are seen in the early iterations, with errors largely minimized at the 10th iteration. Note that the initial mesh comes from UGS.
• Figure 5: Visualization of mask tokens in each iteration. indicates [MASK] tokens, and refers to unmasked tokens.