TOSC: Task-Oriented Shape Completion for Open-World Dexterous Grasp Generation from Partial Point Clouds

TL;DR

Open-world dexterous grasping under partial observations hinges on incomplete geometry; this work introduces Task-Oriented Shape Completion (TOSC) to reconstruct contact regions conditioned on the downstream manipulation task. A three-stage pipeline—foundation-model-based TOSC candidate generation, a 3D discriminative autoencoder for plausibility-based selection and global restoration, and FlowGrasp with constraint-aware flow matching—produces task-aligned grasps from partial point clouds. The approach achieves state-of-the-art performance on task-oriented grasping and shape completion, with significant improvements in Grasp Displacement and Chamfer Distance, and demonstrates strong zero-shot generalization to unseen categories and tasks. These results suggest that task-guided completion, coupled with discriminative selection and constraint-aware grasp synthesis, offers robust and scalable benefits for open-world robotic manipulation.

Abstract

Task-oriented dexterous grasping remains challenging in robotic manipulations of open-world objects under severe partial observation, where significant missing data invalidates generic shape completion. In this paper, to overcome this limitation, we study Task-Oriented Shape Completion, a new task that focuses on completing the potential contact regions rather than the entire shape. We argue that shape completion for grasping should be explicitly guided by the downstream manipulation task. To achieve this, we first generate multiple task-oriented shape completion candidates by leveraging the zero-shot capabilities of object functional understanding from several pre-trained foundation models. A 3D discriminative autoencoder is then proposed to evaluate the plausibility of each generated candidate and optimize the most plausible one from a global perspective. A conditional flow-matching model named FlowGrasp is developed to generate task-oriented dexterous grasps from the optimized shape. Our method achieves state-of-the-art performance in task-oriented dexterous grasping and task-oriented shape completion, improving the Grasp Displacement and the Chamfer Distance over the state-of-the-art by 16.17\% and 55.26%, respectively. In particular, it shows good capabilities in grasping objects with severe missing data. It also demonstrates good generality in handling open-set categories and tasks.

Figures (7)

• Figure 1: By targeting task-oriented shape completion instead of generic shape completion, our method achieves higher completion accuracy, enabling more plausible task-oriented grasps, compared to the baseline wei2024grasp.
• Figure 2: The overview of our method. Taking a partial point cloud of an object, the object's category, and a language description of a manipulation task as input, our method first generates multiple candidates of task-oriented completed shapes by the TOSC candidate generation. It then evaluates the plausibility of the generated candidates and restores the most plausible shape from a global perspective by the TOSC selection and restoration. Last, the task-oriented dexterous grasp is generated by the FlowGrasp.
• Figure 3: The pipeline of the TOSC candidate generation. First, the input point cloud is rendered into a depth map. Then, the ControlNet is adopted to synthesize multiple RGB images using different control scales. Third, the corresponding 3D shapes are then generated with a 3D shape generation network. After segmenting and detecting task-relevant regions in the generated 3D shapes and input point cloud, a point cloud fusion is performed to generate the TOSC candidates.
• Figure 4: Illustrations of the key components in the TOSC selection and restoration.
• Figure 5: The illustrations of the FlowGrasp.