Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Dense Hand-Object(HO) GraspNet with Full Grasping Taxonomy and Dynamics

Woojin Cho, Jihyun Lee, Minjae Yi, Minje Kim, Taeyun Woo, Donghwan Kim, Taewook Ha, Hyokeun Lee, Je-Hwan Ryu, Woontack Woo, Tae-Kyun Kim

TL;DR

This work presents a comprehensive new training dataset for hand-object interaction called HOGraspNet, the only real dataset that captures full grasp taxonomies, providing grasp annotation and wide intraclass variations.

Abstract

Existing datasets for 3D hand-object interaction are limited either in the data cardinality, data variations in interaction scenarios, or the quality of annotations. In this work, we present a comprehensive new training dataset for hand-object interaction called HOGraspNet. It is the only real dataset that captures full grasp taxonomies, providing grasp annotation and wide intraclass variations. Using grasp taxonomies as atomic actions, their space and time combinatorial can represent complex hand activities around objects. We select 22 rigid objects from the YCB dataset and 8 other compound objects using shape and size taxonomies, ensuring coverage of all hand grasp configurations. The dataset includes diverse hand shapes from 99 participants aged 10 to 74, continuous video frames, and a 1.5M RGB-Depth of sparse frames with annotations. It offers labels for 3D hand and object meshes, 3D keypoints, contact maps, and \emph{grasp labels}. Accurate hand and object 3D meshes are obtained by fitting the hand parametric model (MANO) and the hand implicit function (HALO) to multi-view RGBD frames, with the MoCap system only for objects. Note that HALO fitting does not require any parameter tuning, enabling scalability to the dataset's size with comparable accuracy to MANO. We evaluate HOGraspNet on relevant tasks: grasp classification and 3D hand pose estimation. The result shows performance variations based on grasp type and object class, indicating the potential importance of the interaction space captured by our dataset. The provided data aims at learning universal shape priors or foundation models for 3D hand-object interaction. Our dataset and code are available at https://hograspnet2024.github.io/.

Dense Hand-Object(HO) GraspNet with Full Grasping Taxonomy and Dynamics

TL;DR

This work presents a comprehensive new training dataset for hand-object interaction called HOGraspNet, the only real dataset that captures full grasp taxonomies, providing grasp annotation and wide intraclass variations.

Abstract

Existing datasets for 3D hand-object interaction are limited either in the data cardinality, data variations in interaction scenarios, or the quality of annotations. In this work, we present a comprehensive new training dataset for hand-object interaction called HOGraspNet. It is the only real dataset that captures full grasp taxonomies, providing grasp annotation and wide intraclass variations. Using grasp taxonomies as atomic actions, their space and time combinatorial can represent complex hand activities around objects. We select 22 rigid objects from the YCB dataset and 8 other compound objects using shape and size taxonomies, ensuring coverage of all hand grasp configurations. The dataset includes diverse hand shapes from 99 participants aged 10 to 74, continuous video frames, and a 1.5M RGB-Depth of sparse frames with annotations. It offers labels for 3D hand and object meshes, 3D keypoints, contact maps, and \emph{grasp labels}. Accurate hand and object 3D meshes are obtained by fitting the hand parametric model (MANO) and the hand implicit function (HALO) to multi-view RGBD frames, with the MoCap system only for objects. Note that HALO fitting does not require any parameter tuning, enabling scalability to the dataset's size with comparable accuracy to MANO. We evaluate HOGraspNet on relevant tasks: grasp classification and 3D hand pose estimation. The result shows performance variations based on grasp type and object class, indicating the potential importance of the interaction space captured by our dataset. The provided data aims at learning universal shape priors or foundation models for 3D hand-object interaction. Our dataset and code are available at https://hograspnet2024.github.io/.
Paper Structure (20 sections, 1 equation, 11 figures, 4 tables)

This paper contains 20 sections, 1 equation, 11 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Survey on Interaction Datasets
  3. Single-Hand Datasets.
  4. Hand-Object Datasets.
  5. Two-Hand Datasets.
  6. Interacting Human Datasets.
  7. Hand-Object Reconstruction.
  8. HOGraspNet
  9. Dataset Overview
  10. Object Categories and Grasp Taxonomy
  11. Hardware Setup and Data Collection
  12. Data Distributions
  13. MANO and Object Annotation
  14. HALO Annotation
  15. Experimental Results
  16. ...and 5 more sections

Figures (11)

  • Figure 1: (left) Diverse samples in HOGraspNet (best viewed with zoom-in). HOGraspNet captures all hand-object grasp taxonomies with high-quality 3D annotations. (right) Grasp Taxonomy t-SNE. It covers well the grasp taxonomy space with intra-class variations.
  • Figure 2: Structure of HOGraspNet. It captures diverse hand-object grasping at 4 different viewpoints. Example RGB images (A) and depth images (B) are shown, while the fitted hand and object meshes are visualized in (C) and (D). (E) shows the contact map.
  • Figure 3: (left) 33 hand grasping taxonomies, (right) 30 objects used in the dataset. The object types are cylinder, sphere, disk, cuboid, or compound/articulated. They are further dividend to small/medium/large sizes, purporting to cover all grasp taxonomies.
  • Figure 4: (left) Per-object taxonomy examples (right) System setup. The full list is shown in the supplementary.
  • Figure 5: t-SNE van2008visualizing visualization of (left) MANO romero2022embodied shape parameter distributions and (right) grasp feature distributions.
  • ...and 6 more figures