MaskHand: Generative Masked Modeling for Robust Hand Mesh Reconstruction in the Wild

TL;DR

MaskHand tackles monocular 3D hand mesh reconstruction by introducing a generative masked modeling framework that explicitly learns the probabilistic 2D-to-3D mapping. It combines VQ-MANO, which tokenizes 3D hand poses into discrete tokens, with a Context-Guided Masked Transformer that fuses image context, 2D pose cues, and masked token distributions to produce high-confidence hand meshes. A differential masked training objective and confidence-guided iterative sampling enable accurate reconstructions even under occlusions and depth ambiguities, achieving state-of-the-art results on HO3Dv3, FreiHAND, and HInt, and enabling text-conditioned text-to-mesh generation. The approach provides explicit uncertainty estimates for each mesh hypothesis, enhancing robustness and practical applicability in real-world hand understanding tasks and AR/VR applications.

Abstract

Reconstructing a 3D hand mesh from a single RGB image is challenging due to complex articulations, self-occlusions, and depth ambiguities. Traditional discriminative methods, which learn a deterministic mapping from a 2D image to a single 3D mesh, often struggle with the inherent ambiguities in 2D-to-3D mapping. To address this challenge, we propose MaskHand, a novel generative masked model for hand mesh recovery that synthesizes plausible 3D hand meshes by learning and sampling from the probabilistic distribution of the ambiguous 2D-to-3D mapping process. MaskHand consists of two key components: (1) a VQ-MANO, which encodes 3D hand articulations as discrete pose tokens in a latent space, and (2) a Context-Guided Masked Transformer that randomly masks out pose tokens and learns their joint distribution, conditioned on corrupted token sequence, image context, and 2D pose cues. This learned distribution facilitates confidence-guided sampling during inference, producing mesh reconstructions with low uncertainty and high precision. Extensive evaluations on benchmark and real-world datasets demonstrate that MaskHand achieves state-of-the-art accuracy, robustness, and realism in 3D hand mesh reconstruction. Project website: https://m-usamasaleem.github.io/publication/MaskHand/MaskHand.html.

Figures (20)

• Figure 1: MaskHand: a novel generative masked model for accurate and robust 3D hand mesh recovery from single RGB images, excelling in diverse scenarios like occlusions, hand-object interactions, and varied appearances. Watch our Supplemental Video to see it in action!
• Figure 2: MaskHand is a quantifiable probabilistic HMR method that can learn 2D-to-3D mapping distributions and explicitly estimate confidence levels or prediction probabilities for all mesh reconstruction hypotheses
• Figure 3: MaskHand Training Phase. MaskHand consists of two key components: (1) VQ-MANO, which encodes 3D hand poses into a sequence of discrete tokens within a latent space, and (2) a Context-Guided Masked Transformer that models the probabilistic distributions of these tokens, conditioned on the input image, 2D pose cues, and a partially masked token sequence.
• Figure 4: Architecture of Graph-based Anatomical Pose Refinement (GAPR) and Context-Infused Masked Synthesizer, illustrating fusion of anatomical pose dependencies and contextual cues for precise mesh reconstruction.
• Figure 5: MaskHand Inference Phase: Confidence-Guided Iterative Sampling — a step-by-step refinement of pose selection by probabilistically sampling high-confidence tokens.