ShapeGrasp: Zero-Shot Task-Oriented Grasping with Large Language Models through Geometric Decomposition

TL;DR

This work presents a novel zero-shot task-oriented grasping method leveraging a geometric decomposition of the target object into simple, convex shapes that the authors represent in a graph structure, including geometric attributes and spatial relationships.

Abstract

Task-oriented grasping of unfamiliar objects is a necessary skill for robots in dynamic in-home environments. Inspired by the human capability to grasp such objects through intuition about their shape and structure, we present a novel zero-shot task-oriented grasping method leveraging a geometric decomposition of the target object into simple, convex shapes that we represent in a graph structure, including geometric attributes and spatial relationships. Our approach employs minimal essential information - the object's name and the intended task - to facilitate zero-shot task-oriented grasping. We utilize the commonsense reasoning capabilities of large language models to dynamically assign semantic meaning to each decomposed part and subsequently reason over the utility of each part for the intended task. Through extensive experiments on a real-world robotics platform, we demonstrate that our grasping approach's decomposition and reasoning pipeline is capable of selecting the correct part in 92% of the cases and successfully grasping the object in 82% of the tasks we evaluate. Additional videos, experiments, code, and data are available on our project website: https://shapegrasp.github.io/.

Figures (6)

• Figure 1: The ShapeGrasp Pipeline: Given a target object, our RGB+D-based approach decomposes the object into basic convex parts. We propose a heuristic approach to decide which decomposition to use before converting it into a shape graph, allowing an LLM to utilize its commonsense reasoning to identify part semantics and task suitability.
• Figure 2: Different resulting grasps given our shape-based inference pipeline. Parts in orange-boldface are ultimately grasped. Green circles and blue lines represent the part-graph decomposition with each entity's associated attributes.
• Figure 3: ShapeGrasp Prompting: Given a geometric decomposition graph, we infer a suitable grasping point through a chain of four consecutive prompts: two for semantic part identification, and two for task-oriented reasoning and selection. Ablations of these prompts can be found in Table \ref{['tab:ablation_study']}.
• Figure 4: Overview of the $38$ objects used in our study, inspired by the objects introduced in LERF-TOGO lerftogo2023.
• Figure 5: Threshold analysis for 2D (top)/3D (bottom) decompositions on sunglasses (complex) and screwdriver (simple).