GraspSAM: When Segment Anything Model Meets Grasp Detection

TL;DR

This paper introduces GraspSAM, an innovative extension of the Segment Anything Model (SAM) designed for prompt-driven and category-agnostic grasp detection, which achieves state-of-the-art (SOTA) performance across multiple datasets, including Jacquard, Grasp-Anything, and Grasp-Anything++.

Abstract

Grasp detection requires flexibility to handle objects of various shapes without relying on prior knowledge of the object, while also offering intuitive, user-guided control. This paper introduces GraspSAM, an innovative extension of the Segment Anything Model (SAM), designed for prompt-driven and category-agnostic grasp detection. Unlike previous methods, which are often limited by small-scale training data, GraspSAM leverages the large-scale training and prompt-based segmentation capabilities of SAM to efficiently support both target-object and category-agnostic grasping. By utilizing adapters, learnable token embeddings, and a lightweight modified decoder, GraspSAM requires minimal fine-tuning to integrate object segmentation and grasp prediction into a unified framework. The model achieves state-of-the-art (SOTA) performance across multiple datasets, including Jacquard, Grasp-Anything, and Grasp-Anything++. Extensive experiments demonstrate the flexibility of GraspSAM in handling different types of prompts (such as points, boxes, and language), highlighting its robustness and effectiveness in real-world robotic applications.

Figures (6)

• Figure 1: Conventional methods use separate networks for object identification and grasp prediction, while GraspSAM (Ours) predicts both the object mask and grasps from a single RGB image and prompt in a single step.
• Figure 2: GraspSAM overall pipeline. GraspSAM builds upon the zero-shot capabilities of the SAM by adding an adapter for the image encoder, a decoder with several additional MLP layers, and lightweight token learning to enable object amsk and grasp map prediction. During training, the weights of the SAM modules are freezed, and only the adapter and the MLP layers in the decoder are updated. The learnable tokens consist of embedded token from prompts such as points or boxes obtained via a) mouse clicks, eye-gazing, or b) Grounding-DINO (G.D) and learnable tokens used to predict the object mask and grasp map.
• Figure 3: Visualization of GraspSAM prediction. We visualize the input prompts (10 points) along with the predicted outputs, including the object mask and grasp box. Additionally, we display the predicted grasp quality map. (a) corresponds to the Grasp-Anything Base set, while (b) represents the New set.
• Figure 4: Visualization of in-the-wild grasp detection results
• Figure 5: Real-world experiments settings.