Controllable Hand Grasp Generation for HOI and Efficient Evaluation Methods

TL;DR

The paper tackles the challenge of generating controllable hand grasps for HOI using only 2D information, addressing the reliance on 3D geometry and lack of placement control in prior work. It introduces higher-order representations (HOR) of the hand pose and a diffusion-based generator with a cross-attention decoder to place grasps at user-defined locations around objects, guided by HOR descriptors. It further proposes an HOR-based f-FID evaluation framework to provide efficient and unbiased metrics for hand pose generation. Through extensive experiments on HOI4D, the approach shows improved pose quality, better controllability, and robust evaluation compared with traditional metrics like FID and MMD. The work offers a scalable pathway for generating realistic HOI data without full 3D geometry and highlights HORs as a flexible plug-in for both generation and evaluation tasks.

Abstract

Controllable affordance Hand-Object Interaction (HOI) generation has become an increasingly important area of research in computer vision. In HOI generation, the hand grasp generation is a crucial step for effectively controlling the geometry of the hand. Current hand grasp generation methods rely on 3D information for both the hand and the object. In addition, these methods lack controllability concerning the hand's location and orientation. We treat the hand pose as the discrete graph structure and exploit the geometric priors. It is well established that higher order contextual dependency among the points improves the quality of the results in general. We propose a framework of higher order geometric representations (HOR's) inspired by spectral graph theory and vector algebra to improve the quality of generated hand poses. We demonstrate the effectiveness of our proposed HOR's in devising a controllable novel diffusion method (based on 2D information) for hand grasp generation that outperforms the state of the art (SOTA). Overcoming the limitations of existing methods: like lacking of controllability and dependency on 3D information. Once we have the generated pose, it is very natural to evaluate them using a metric. Popular metrics like FID and MMD are biased and inefficient for evaluating the generated hand poses. Using our proposed HOR's, we introduce an efficient and stable framework of evaluation metrics for grasp generation methods, addressing inefficiencies and biases in FID and MMD.

Figures (4)

• Figure 1: Block (A), is the overview of our proposed controllable hand pose generation framework. In block (B), we present our position aware cross attention decoder. Block (C), is our proposed HOR's.
• Figure 2: Qualitative comparison of our proposed method with the baseline method. The first column shows the Input images, and the second column shows the Mask images. The third column presents the results from the vanilla model (baseline). The fourth and fifth columns display the results from the PoseNet and PoseNetD models, respectively. Subsequent columns show the results of our proposed methods under different losses used while training. Each row corresponds to a different test image, demonstrating the performance of each method across a variety of input scenarios. Second row for each object shows the generated hand pose results from out of distribution orientation. We can see that our proposed method outperforms the vanilla method (columns 3,4). Using HOR based loss with proposed method outperforms the proposed method with Identity or no reconstruction loss, on qualitative results.
• Figure 3: In this figure, we analyze the alignment between qualitative and quantitative results of various evaluation metrics. The numbers in each row correspond to evaluation metrics. Notably, row $2$ demonstrates that the DenseT metric aligns well with the generated poses, low value for good quality and high value for poor quality. Conversely, the Identity metric does not; the value in column $1$$(0.45)$ is lower than that in column $4$$(0.67)$, whereas it should be higher based on the qualitative alignment. Rows $4$ and $5$ present the quantitative results for FID computed on grayscale and colored images denoted as Vanilla-B and Vanilla-C. We can see that Vanilla-C aligns well with qualitative results then Vanilla-B, showing that FID is sensitive to rendering color choice, which impacts its stability.
• Figure 4: Grabnet hand grasp generation result for various objects in respective order from \ref{['fig:qualitative_res_all']}.