Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

HGGT: Robust and Flexible 3D Hand Mesh Reconstruction from Uncalibrated Images

Yumeng Liu, Xiao-Xiao Long, Marc Habermann, Xuanze Yang, Cheng Lin, Yuan Liu, Yuexin Ma, Wenping Wang, Ligang Liu

Abstract

Recovering high-fidelity 3D hand geometry from images is a critical task in computer vision, holding significant value for domains such as robotics, animation and VR/AR. Crucially, scalable applications demand both accuracy and deployment flexibility, requiring the ability to leverage massive amounts of unstructured image data from the internet or enable deployment on consumer-grade RGB cameras without complex calibration. However, current methods face a dilemma. While single-view approaches are easy to deploy, they suffer from depth ambiguity and occlusion. Conversely, multi-view systems resolve these uncertainties but typically demand fixed, calibrated setups, limiting their real-world utility. To bridge this gap, we draw inspiration from 3D foundation models that learn explicit geometry directly from visual data. By reformulating hand reconstruction from arbitrary views as a visual-geometry grounded task, we propose a feed-forward architecture that, for the first time in literature, jointly infers 3D hand meshes and camera poses from uncalibrated views. Extensive evaluations show that our approach outperforms state-of-the-art benchmarks and demonstrates strong generalization to uncalibrated, in-the-wild scenarios. Here is the link of our project page: https://lym29.github.io/HGGT/.

HGGT: Robust and Flexible 3D Hand Mesh Reconstruction from Uncalibrated Images

Abstract

Recovering high-fidelity 3D hand geometry from images is a critical task in computer vision, holding significant value for domains such as robotics, animation and VR/AR. Crucially, scalable applications demand both accuracy and deployment flexibility, requiring the ability to leverage massive amounts of unstructured image data from the internet or enable deployment on consumer-grade RGB cameras without complex calibration. However, current methods face a dilemma. While single-view approaches are easy to deploy, they suffer from depth ambiguity and occlusion. Conversely, multi-view systems resolve these uncertainties but typically demand fixed, calibrated setups, limiting their real-world utility. To bridge this gap, we draw inspiration from 3D foundation models that learn explicit geometry directly from visual data. By reformulating hand reconstruction from arbitrary views as a visual-geometry grounded task, we propose a feed-forward architecture that, for the first time in literature, jointly infers 3D hand meshes and camera poses from uncalibrated views. Extensive evaluations show that our approach outperforms state-of-the-art benchmarks and demonstrates strong generalization to uncalibrated, in-the-wild scenarios. Here is the link of our project page: https://lym29.github.io/HGGT/.
Paper Structure (43 sections, 3 equations, 17 figures, 3 tables)

This paper contains 43 sections, 3 equations, 17 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Multi-view 3D Reconstruction.
  4. Hand Reconstruction.
  5. Method
  6. Problem Definition
  7. Dataset Preparation
  8. Model Design
  9. Multi-View Feature Aggregation.
  10. Unified Cross-Attention Refinement.
  11. Parametric Hand Head.
  12. Camera Pose Estimation Head
  13. Loss Function
  14. Hand Loss.
  15. Camera Loss.
  16. ...and 28 more sections

Figures (17)

  • Figure 1: We introduce Hand Geometry Grounding Transformer (HGGT), a scalable and generalized solution for 3D hand mesh recovery. Our method unifies diverse data sources to achieve robust performance across varying camera viewpoints and environments.
  • Figure 2: The pipeline of HGGT. Given uncalibrated multi-view images, we first employ a VGGT Aggregator to extract image tokens and initial camera tokens. These are processed alongside learnable random hand tokens via a series of Cross-attention Blocks. Finally, two parallel heads predict the camera parameters and the canonical MANO parameters ($\boldsymbol{\theta}, \boldsymbol{\beta}, \mathbf{t}$), which can be re-projected onto the input views for verification.
  • Figure 3: Samples from our synthetic dataset. It contains diverse photorealistic hand-object interactions.
  • Figure 4: Camera Layout Visualization.
  • Figure 5: Failure cases of off-the-shelf VGGT on HO3D datasets.
  • ...and 12 more figures