AffordGrasp: In-Context Affordance Reasoning for Open-Vocabulary Task-Oriented Grasping in Clutter

TL;DR

AffordGrasp tackles open-vocabulary task-oriented grasping in clutter by integrating in-context reasoning from Vision-Language Models with open-vocabulary visual grounding and grasp synthesis. It uses GPT-4o to extract task goals, identify relevant objects, and determine optimal parts and affordances, which are then grounded at the pixel level via VLPart and used by AnyGrasp to generate 6D grasp poses. The framework operates without task-specific training data and demonstrates state-of-the-art performance in both simulated and real-world clutter, outperforming baselines across a range of objects. This approach advances embodied AI by enabling intuitive human-robot interaction and robust manipulation in unconstrained environments, with broad implications for flexible robotic assistants and automation.

Abstract

Inferring the affordance of an object and grasping it in a task-oriented manner is crucial for robots to successfully complete manipulation tasks. Affordance indicates where and how to grasp an object by taking its functionality into account, serving as the foundation for effective task-oriented grasping. However, current task-oriented methods often depend on extensive training data that is confined to specific tasks and objects, making it difficult to generalize to novel objects and complex scenes. In this paper, we introduce AffordGrasp, a novel open-vocabulary grasping framework that leverages the reasoning capabilities of vision-language models (VLMs) for in-context affordance reasoning. Unlike existing methods that rely on explicit task and object specifications, our approach infers tasks directly from implicit user instructions, enabling more intuitive and seamless human-robot interaction in everyday scenarios. Building on the reasoning outcomes, our framework identifies task-relevant objects and grounds their part-level affordances using a visual grounding module. This allows us to generate task-oriented grasp poses precisely within the affordance regions of the object, ensuring both functional and context-aware robotic manipulation. Extensive experiments demonstrate that AffordGrasp achieves state-of-the-art performance in both simulation and real-world scenarios, highlighting the effectiveness of our method. We believe our approach advances robotic manipulation techniques and contributes to the broader field of embodied AI. Project website: https://eqcy.github.io/affordgrasp/.

Figures (6)

• Figure 1: Compared to existing task-oriented grasping methods, our approach offers three key advantages: (1) leveraging VLMs for affordance reasoning, enabling better understanding of user intentions from implicit language instructions and visual scene observations; (2) a training-free pipeline, eliminating the need for annotated data to train task-oriented grasp evaluators; and (3) the ability to handle cluttered scenes, rather than being limited to single-object grasping.
• Figure 2: Overall Framework of AffordGrasp. The framework processes user instructions and RGB-D scene observations to achieve open-vocabulary task-oriented grasping in clutter. We leverage GPT-4o hurst2024gpt for in-context affordance reasoning, decomposing the process into three steps: (1) Extracting the task goal and functional requirements from implicit user instructions (e.g., "I want to scoop something"). (2) Identifying the most task-relevant object in the RGB image (e.g., a wooden spoon). (3) Decomposing the object into functional parts and selecting the optimal graspable part (e.g., the handle) based on its affordances. Based on the reasoning results, a visual affordance grounding module grounds the inferred object and part affordances into pixel-level masks. With the affordance mask and RGB-D images, we employ AnyGrasp 10167687 to generate task-oriented 6D grasp poses on the target part.
• Figure 3: The prompt for in-context affordance reasoning is designed, and an example is provided for illustration.
• Figure 4: Simulation Cases of Grasping in Clutter: The affordances of the target object are indicated with red stars.
• Figure 5: Real-world Examples of Grasping in Clutter: The visualization of affordance grounding and task-oriented grasp generation are provided.