Attention-Guided Integration of CLIP and SAM for Precise Object Masking in Robotic Manipulation

TL;DR

This work tackles the challenge of precise object masking for robotic manipulation in a convenience-store domain, where domain-specific data scarcity hampers general pipelines. It introduces a gradient-attention guided pipeline that fuses CLIP's multimodal embeddings with SAM's segmentation, aided by Grad-CAM and domain-specific fine-tuning. Key contributions include a domain-focused dataset and CLIP fine-tuning, Grad-CAM guided SAM prompting, and empirical IoU gains demonstrating improved masking accuracy. The approach enables more reliable and adaptive manipulation of retail products in structured environments and informs the design of multimodal perception systems for real-world robotics.

Abstract

This paper introduces a novel pipeline to enhance the precision of object masking for robotic manipulation within the specific domain of masking products in convenience stores. The approach integrates two advanced AI models, CLIP and SAM, focusing on their synergistic combination and the effective use of multimodal data (image and text). Emphasis is placed on utilizing gradient-based attention mechanisms and customized datasets to fine-tune performance. While CLIP, SAM, and Grad- CAM are established components, their integration within this structured pipeline represents a significant contribution to the field. The resulting segmented masks, generated through this combined approach, can be effectively utilized as inputs for robotic systems, enabling more precise and adaptive object manipulation in the context of convenience store products.

Figures (6)

• Figure 1: A convenience store scene displaying various products used for segmentation tasks in our laboratory.
• Figure 2: Summary of our system integration. The diagram illustrates the flow from input processing in CLIP to the generation of accurate segmentation masks by SAM, guided by Grad-CAM attention maps.
• Figure 3: Top predictions and confidence scores for four different images. Each image is shown with a caption listing the top predictions generated by the model.
• Figure 4: Attention maps highlighting different parts of the object: (a) Top Part, (b) Bottom Part, and (c) Side Part. These visualizations show where the CLIP model focuses, providing crucial guidance for the segmentation process.
• Figure 5: Different segmentation prompts: (a) Single Point, (b) Multiple Points, and (c) Bounding Box, each guide the segmentation process in the SAM model to accurately identify objects.