Reasoning Grasping via Multimodal Large Language Model

TL;DR

This work tackles implicit instruction-driven robotic grasping by introducing the task of reasoning grasping and an end-to-end framework that fuses a multimodal LLM with a vision-based grasping detector. The model leverages a LoRA-finetuned LLaVA to interpret both text and images, uses a [SPT] strategy to identify grasp targets, and generates grasp poses $g=(x,y,\theta,w,q)$ through joint training with a grasping objective. An extended GraspNet-1B-derived dataset provides 1,730 reasoning instructions, 64 objects, 109 parts, and ~100 million grasps, enabling evaluation in both cluttered scenes and real-world settings. Real-world experiments show our approach surpasses modular baselines, achieving high token-target accuracy (39/40) and more successful grasps, highlighting the practical potential for interpreting implicit human intents in assistive robotics.

Abstract

Despite significant progress in robotic systems for operation within human-centric environments, existing models still heavily rely on explicit human commands to identify and manipulate specific objects. This limits their effectiveness in environments where understanding and acting on implicit human intentions are crucial. In this study, we introduce a novel task: reasoning grasping, where robots need to generate grasp poses based on indirect verbal instructions or intentions. To accomplish this, we propose an end-to-end reasoning grasping model that integrates a multimodal Large Language Model (LLM) with a vision-based robotic grasping framework. In addition, we present the first reasoning grasping benchmark dataset generated from the GraspNet-1 billion, incorporating implicit instructions for object-level and part-level grasping. Our results show that directly integrating CLIP or LLaVA with the grasp detection model performs poorly on the challenging reasoning grasping tasks, while our proposed model demonstrates significantly enhanced performance both in the reasoning grasping benchmark and real-world experiments.

Figures (9)

• Figure 1: Overview. We integrate the reasoning abilities of multi-modal Large Language Models with robotic grasping. The resulting model interprets complex and implicit instructions, accurately predicts robotic grasping poses for target objects or specific parts within cluttered environments, and supports multi-round conversations with users. In the textual output from the model, the grasping target is indicated by two special tokens [SPT], as demonstrated in the figure. The output grasp poses are visualized in the images with rectangles.
• Figure 2: Framework of the proposed reasoning grasping model. This model processes visual images $v$ and textual instructions $t$ to output the grasp pose $g$ for the specified target object or part. The embeddings of the grasp target are passed to the grasping module for grasping poses detection.
• Figure 3: Real world experiments with implicit instructions.
• Figure 4: Comparison of our method with GraspGPT and LAN-grasp.
• Figure 5: Grasping detection with few-shot prompting using GPT-4v on Cornell Grasp dataset.