PartDexTOG: Generating Dexterous Task-Oriented Grasping via Language-driven Part Analysis

TL;DR

PartDexTOG addresses dexterous task-oriented grasping in open-world settings by leveraging language-driven part analysis. It generates category-level and part-level grasp descriptions with LLM prompts and grounds per-part grasps via a category-part conditioned diffusion model, parameterized by the 61-DOF hand vector $g \in \mathbb{R}^{61}$. Multi-scale part segmentation with PartSLIP and cross-attention fuse language and geometry to enable zero-shot generalization and robust performance on novel categories, as demonstrated on OakInk-shape with state-of-the-art metrics (including P-FID and LLM-assisted scores). This approach represents a scalable pathway to dexterous manipulation by entwining open-world knowledge with fine-grained part reasoning for grasp synthesis.

Abstract

Task-oriented grasping is a crucial yet challenging task in robotic manipulation. Despite the recent progress, few existing methods address task-oriented grasping with dexterous hands. Dexterous hands provide better precision and versatility, enabling robots to perform task-oriented grasping more effectively. In this paper, we argue that part analysis can enhance dexterous grasping by providing detailed information about the object's functionality. We propose PartDexTOG, a method that generates dexterous task-oriented grasps via language-driven part analysis. Taking a 3D object and a manipulation task represented by language as input, the method first generates the category-level and part-level grasp descriptions w.r.t the manipulation task by LLMs. Then, a category-part conditional diffusion model is developed to generate a dexterous grasp for each part, respectively, based on the generated descriptions. To select the most plausible combination of grasp and corresponding part from the generated ones, we propose a measure of geometric consistency between grasp and part. We show that our method greatly benefits from the open-world knowledge reasoning on object parts by LLMs, which naturally facilitates the learning of grasp generation on objects with different geometry and for different manipulation tasks. Our method ranks top on the OakInk-shape dataset over all previous methods, improving the Penetration Volume, the Grasp Displace, and the P-FID over the state-of-the-art by $3.58\%$, $2.87\%$, and $41.43\%$, respectively. Notably, it demonstrates good generality in handling novel categories and tasks.

Figures (13)

• Figure 1: PartDexTOG generates high-quality dexterous task-oriented grasps given a language instruction with the object category and a manipulation task. It selects the most plausible part of the target object to grasp, thanks to the open-world knowledge reasoning capabilities of LLMs.
• Figure 2: The overview of PartDexTOG. Taking a 3D object and a manipulation task as inputs, the method first generates the category-level grasp descriptions by LLMs. Then, the potential multi-scale part labels are generated based on the object category. These part labels are segmented using PartSlip to identify the valid point sets. The part-level grasp description for each valid part is generated using LLMs. Last, the features of the descriptions are aggregated via cross-attention. The language features, together with the point cloud features, are fed into a conditional diffusion model to generate a dexterous grasp for each valid part, respectively. The most plausible combination of grasp and the corresponding part is selected with a measure of geometric consistency between grasp and part.
• Figure 3: The LLM prompts utilized in PartDexTOG.
• Figure 4: Our method selects the most plausible combination of grasp and the corresponding part from the generated ones by measuring the geometric consistency between hand and part.
• Figure 5: Visual comparison of the generated task-oriented grasps by PartDexTOG and the baselines.