Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Joint Diffusion for Universal Hand-Object Grasp Generation

Jinkun Cao, Jingyuan Liu, Kris Kitani, Yi Zhou

TL;DR

Joint Hand-Object Diffusion (JHOD) presents a unified latent-diffusion approach to generate 3D hand grasps and object shapes either jointly or conditioned on an object. By encoding hand and object modalities into separate latent streams and employing asynchronous denoising schedulers, the model can leverage heterogeneous datasets, including large-scale object shapes from ShapeNet, to learn a robust object prior and generalize to unseen geometries. The method demonstrates strong performance in both conditional and unconditional grasp generation, with ablations showing benefits from additional object data and scheduler decoupling. This work advances universal grasp generation and provides a flexible, data-efficient framework for hand-object synthesis with potential downstream use in image synthesis and robotics planning.

Abstract

Predicting and generating human hand grasp over objects is critical for animation and robotic tasks. In this work, we focus on generating both the hand and objects in a grasp by a single diffusion model. Our proposed Joint Hand-Object Diffusion (JHOD) models the hand and object in a unified latent representation. It uses the hand-object grasping data to learn to accommodate hand and object to form plausible grasps. Also, to enforce the generalizability over diverse object shapes, it leverages large-scale object datasets to learn an inclusive object latent embedding. With or without a given object as an optional condition, the diffusion model can generate grasps unconditionally or conditional to the object. Compared to the usual practice of learning object-conditioned grasp generation from only hand-object grasp data, our method benefits from more diverse object data used for training to handle grasp generation more universally. According to both qualitative and quantitative experiments, both conditional and unconditional generation of hand grasp achieves good visual plausibility and diversity. With the extra inclusiveness of object representation learned from large-scale object datasets, the proposed method generalizes well to unseen object shapes.

Joint Diffusion for Universal Hand-Object Grasp Generation

TL;DR

Joint Hand-Object Diffusion (JHOD) presents a unified latent-diffusion approach to generate 3D hand grasps and object shapes either jointly or conditioned on an object. By encoding hand and object modalities into separate latent streams and employing asynchronous denoising schedulers, the model can leverage heterogeneous datasets, including large-scale object shapes from ShapeNet, to learn a robust object prior and generalize to unseen geometries. The method demonstrates strong performance in both conditional and unconditional grasp generation, with ablations showing benefits from additional object data and scheduler decoupling. This work advances universal grasp generation and provides a flexible, data-efficient framework for hand-object synthesis with potential downstream use in image synthesis and robotics planning.

Abstract

Predicting and generating human hand grasp over objects is critical for animation and robotic tasks. In this work, we focus on generating both the hand and objects in a grasp by a single diffusion model. Our proposed Joint Hand-Object Diffusion (JHOD) models the hand and object in a unified latent representation. It uses the hand-object grasping data to learn to accommodate hand and object to form plausible grasps. Also, to enforce the generalizability over diverse object shapes, it leverages large-scale object datasets to learn an inclusive object latent embedding. With or without a given object as an optional condition, the diffusion model can generate grasps unconditionally or conditional to the object. Compared to the usual practice of learning object-conditioned grasp generation from only hand-object grasp data, our method benefits from more diverse object data used for training to handle grasp generation more universally. According to both qualitative and quantitative experiments, both conditional and unconditional generation of hand grasp achieves good visual plausibility and diversity. With the extra inclusiveness of object representation learned from large-scale object datasets, the proposed method generalizes well to unseen object shapes.
Paper Structure (24 sections, 17 equations, 12 figures, 5 tables)

This paper contains 24 sections, 17 equations, 12 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Related Works
  3. Method
  4. Problem Formulation
  5. Preliminaries
  6. Latent Diffusion
  7. Modality Representations
  8. Object Generation
  9. Joint Hand-Object Diffusion (JHOD)
  10. Latent Codes for Different Modalities
  11. Asynchronous Denoising Schedulers
  12. Training and Sampling
  13. Experiments
  14. Setups
  15. Object-Conditioned Grasp Generation
  16. ...and 9 more sections

Figures (12)

  • Figure 1: Applications of our proposed method Joint Hand-Object Diffusion (JHOD). Left: generating the hand grasp on an unseen object. Middle: jointly sampling a combination of the hand and the object for a plausible grasp. Right: the generated grasp can be used as guidance for generating photo-realistic grasping images using existing image generation tools firefly.
  • Figure 2: The illustration of our proposed Joint Hand-Object Diffusion (JHOD). We present the main modules of the model while removing the secondary modules for simplicity. The optional object condition can turn the generation into object-conditioned.
  • Figure 3: Generated samples on objects from GRAB taheri2020grab dataset.
  • Figure 4: Generated samples on objects from OakInk yang2022oakink dataset.
  • Figure 5: Samples of the unconditional generation of hand grasp together with objects.
  • ...and 7 more figures