Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Web2Grasp: Learning Functional Grasps from Web Images of Hand-Object Interactions

Hongyi Chen, Yunchao Yao, Yufei Ye, Zhixuan Xu, Homanga Bharadhwaj, Jiashun Wang, Shubham Tulsiani, Zackory Erickson, Jeffrey Ichnowski

TL;DR

The paper addresses the challenge of learning functional grasps for dexterous hands by exploiting 3D reconstructed hand-object interactions from web images, bypassing costly demonstrations. It builds a pipeline that retargets human HOI to the ShadowHand, aligns object meshes, and trains an interaction-centric DRO model to predict a dense distance matrix for grasp generation, with simulator augmentation to prune and extend data. The approach achieves strong simulation performance (61.8% on web data, 83.4% with sim-aug) and robust sim-to-real transfer (85% on LEAP Hand) across challenging objects, outperforming baselines. This work demonstrates a scalable, data-efficient path to functional grasping and highlights practical considerations for improving HOI quality and extending to broader task distributions.

Abstract

Functional grasp is essential for enabling dexterous multi-finger robot hands to manipulate objects effectively. However, most prior work either focuses on power grasping, which simply involves holding an object still, or relies on costly teleoperated robot demonstrations to teach robots how to grasp each object functionally. Instead, we propose extracting human grasp information from web images since they depict natural and functional object interactions, thereby bypassing the need for curated demonstrations. We reconstruct human hand-object interaction (HOI) 3D meshes from RGB images, retarget the human hand to multi-finger robot hands, and align the noisy object mesh with its accurate 3D shape. We show that these relatively low-quality HOI data from inexpensive web sources can effectively train a functional grasping model. To further expand the grasp dataset for seen and unseen objects, we use the initially-trained grasping policy with web data in the IsaacGym simulator to generate physically feasible grasps while preserving functionality. We train the grasping model on 10 object categories and evaluate it on 9 unseen objects, including challenging items such as syringes, pens, spray bottles, and tongs, which are underrepresented in existing datasets. The model trained on the web HOI dataset, achieving a 75.8% success rate on seen objects and 61.8% across all objects in simulation, with a 6.7% improvement in success rate and a 1.8x increase in functionality ratings over baselines. Simulator-augmented data further boosts performance from 61.8% to 83.4%. The sim-to-real transfer to the LEAP Hand achieves a 85% success rate. Project website is at: https://web2grasp.github.io/.

Web2Grasp: Learning Functional Grasps from Web Images of Hand-Object Interactions

TL;DR

The paper addresses the challenge of learning functional grasps for dexterous hands by exploiting 3D reconstructed hand-object interactions from web images, bypassing costly demonstrations. It builds a pipeline that retargets human HOI to the ShadowHand, aligns object meshes, and trains an interaction-centric DRO model to predict a dense distance matrix for grasp generation, with simulator augmentation to prune and extend data. The approach achieves strong simulation performance (61.8% on web data, 83.4% with sim-aug) and robust sim-to-real transfer (85% on LEAP Hand) across challenging objects, outperforming baselines. This work demonstrates a scalable, data-efficient path to functional grasping and highlights practical considerations for improving HOI quality and extending to broader task distributions.

Abstract

Functional grasp is essential for enabling dexterous multi-finger robot hands to manipulate objects effectively. However, most prior work either focuses on power grasping, which simply involves holding an object still, or relies on costly teleoperated robot demonstrations to teach robots how to grasp each object functionally. Instead, we propose extracting human grasp information from web images since they depict natural and functional object interactions, thereby bypassing the need for curated demonstrations. We reconstruct human hand-object interaction (HOI) 3D meshes from RGB images, retarget the human hand to multi-finger robot hands, and align the noisy object mesh with its accurate 3D shape. We show that these relatively low-quality HOI data from inexpensive web sources can effectively train a functional grasping model. To further expand the grasp dataset for seen and unseen objects, we use the initially-trained grasping policy with web data in the IsaacGym simulator to generate physically feasible grasps while preserving functionality. We train the grasping model on 10 object categories and evaluate it on 9 unseen objects, including challenging items such as syringes, pens, spray bottles, and tongs, which are underrepresented in existing datasets. The model trained on the web HOI dataset, achieving a 75.8% success rate on seen objects and 61.8% across all objects in simulation, with a 6.7% improvement in success rate and a 1.8x increase in functionality ratings over baselines. Simulator-augmented data further boosts performance from 61.8% to 83.4%. The sim-to-real transfer to the LEAP Hand achieves a 85% success rate. Project website is at: https://web2grasp.github.io/.
Paper Structure (20 sections, 9 figures, 5 tables)

This paper contains 20 sections, 9 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Dexterous Robotic Grasping and Functional Grasping
  4. Human Hand-Object Interaction from Images and Videos.
  5. Method
  6. Reconstructing Robotic Hand-Object Interaction
  7. Interaction-Centric Grasping Model Training
  8. Simulator-Augmented Dataset
  9. Experiments
  10. HOI Reconstruction Results
  11. Simulation Grasp Experiment
  12. Real World Experiment
  13. Conclusion
  14. Simulation Experimental Details
  15. Grasping Model Design and Training
  16. ...and 5 more sections

Figures (9)

  • Figure 1: Top: Web images and reconstructed HOI grasps. Bottom: Functionally valid grasps in simulation and in the real-world.
  • Figure 2: Hand-Object Interaction (HOI) Collection, Grasp Model Training and Execution Pipeline.Web2Grasp (a) collects images of humans grasping objects from the web, the (b) uses HOI reconstruction to produce a functional grasp dataset, potentially containing penetrations and unrealistic contacts. (c): Web2Grasp trains a DRO grasping model on the HOI dataset to predict target joint configurations for grasp execution. (d): Web2Grasp uses simulation to collect physically feasible grasps to expand the dataset and retrain the model. The dashed arrowed line indicates input data for DRO model training.
  • Figure 3: Visualization of HOI Reconstruction from Web Images. Reconstructions for representative objects. Success cases---Row 1: Power Drill, Pen; Row 2: Spray Bottle, Wine Glass; Row 3: Syringe, Tongs. Failure cases---Row 4: Lantern, Scissors. From left to right: original web image, web image overlaid with human HOI mesh, human HOI mesh alone, and robot HOI mesh with retargeted robot hand and aligned object.
  • Figure 4: Visualization of Generated Grasps Across All Objects. The first row shows objects seen during web-data training, while the second row presents unseen objects from the test set.
  • Figure 5: Failed Grasps Without Friction and Softness from Grip Tape. Row 1: Spray Bottle; Row 2: Syringe.
  • ...and 4 more figures