Decomposed Vector-Quantized Variational Autoencoder for Human Grasp Generation

TL;DR

The paper addresses realistic human grasp generation by overcoming limitations of whole-hand encodings through a Decomposed Vector-Quantized VAE (DVQ-VAE) that encodes the hand into six parts and the object into discrete priors. It introduces two object encoders (type and pose) and a two-stage autoregressive decoding (posture then position) guided by skeletal constraints, implemented with multiple codebooks and PixelCNN. The approach yields improvements in penetration, stability, diversity, and speed across four benchmarks, with a notable quality-index gain and strong out-of-domain generalization. This work enables more controllable, diverse, and physically plausible grasps, benefiting robotics, VR/AR, and animation applications. Technical contributions include the part-aware decomposed architecture, dual-stage decoding, and a comprehensive loss combining discrete latent learning, morphology constraints, and contact realism.

Abstract

Generating realistic human grasps is a crucial yet challenging task for applications involving object manipulation in computer graphics and robotics. Existing methods often struggle with generating fine-grained realistic human grasps that ensure all fingers effectively interact with objects, as they focus on encoding hand with the whole representation and then estimating both hand posture and position in a single step. In this paper, we propose a novel Decomposed Vector-Quantized Variational Autoencoder (DVQ-VAE) to address this limitation by decomposing hand into several distinct parts and encoding them separately. This part-aware decomposed architecture facilitates more precise management of the interaction between each component of hand and object, enhancing the overall reality of generated human grasps. Furthermore, we design a newly dual-stage decoding strategy, by first determining the type of grasping under skeletal physical constraints, and then identifying the location of the grasp, which can greatly improve the verisimilitude as well as adaptability of the model to unseen hand-object interaction. In experiments, our model achieved about 14.1% relative improvement in the quality index compared to the state-of-the-art methods in four widely-adopted benchmarks. Our source code is available at https://github.com/florasion/D-VQVAE.

Figures (10)

• Figure 1: Illustration of the proposed grasp generation model. Initially, we utilize Decomposed VQ-VAE (DVQ-VAE) to learn the prior distributions of the object and each hand component (i.e., five fingers and the palm) during training. Specifically, we divide the decoding process into two stages hand posture and position generation. During inference, we perform autoregression using the object as a guide to obtain the realistically generated human grasp.
• Figure 2: Overall architecture of the proposed DVQ-VAE model which is based on the encoder-decoder paradigm. During training, the model takes both hand vertices and object point clouds as inputs and maps them into discrete latent spaces consisting of seven codebooks (i.e., one for object and six for hand) based on different hand components to generate hands. While, at the inference phase, we only use object point clouds as input to generate hands capable of grasping the given object.
• Figure 3: Performance comparison of our method and other models in high-quality ratio w.r.t the penetration threshold for different models on the HO-3D dataset.
• Figure 4: The number of indices used in each codebook for our DVQ-VAE.
• Figure 5: The percentage of each score of each model in human evaluation.