Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Show and Grasp: Few-shot Semantic Segmentation for Robot Grasping through Zero-shot Foundation Models

Leonardo Barcellona, Alberto Bacchin, Matteo Terreran, Emanuele Menegatti, Stefano Ghidoni

TL;DR

The paper addresses the limited generalization of few-shot semantic segmentation for robotic grasping by integrating zero-shot foundation models (Segment Anything Model and CutLER) with a high-performing Few-Shot Classifier (PMF). The method inverts the standard FSC role, using the zero-shot segmentator to generate candidate masks and the classifier to select the mask most similar to the support, with the resulting semantic heatmap guiding a grasp synthesis network that outputs a 4-DoF grasp. The approach achieves state-of-the-art results on Graspnet-1B (+10.5% mIoU) and OCID-grasp (+1.6% AP), plus notable real-world gains in grasp accuracy (+21.7%), all without any fine-tuning. Experiments on 47 real objects demonstrate robustness to appearance changes and instance-level vs. class-level generalization, while the paper also discusses limitations and future directions such as 7-DoF grasps and integration with Visual-Language Models for further gains.

Abstract

The ability of a robot to pick an object, known as robot grasping, is crucial for several applications, such as assembly or sorting. In such tasks, selecting the right target to pick is as essential as inferring a correct configuration of the gripper. A common solution to this problem relies on semantic segmentation models, which often show poor generalization to unseen objects and require considerable time and massive data to be trained. To reduce the need for large datasets, some grasping pipelines exploit few-shot semantic segmentation models, which are capable of recognizing new classes given a few examples. However, this often comes at the cost of limited performance and fine-tuning is required to be effective in robot grasping scenarios. In this work, we propose to overcome all these limitations by combining the impressive generalization capability reached by foundation models with a high-performing few-shot classifier, working as a score function to select the segmentation that is closer to the support set. The proposed model is designed to be embedded in a grasp synthesis pipeline. The extensive experiments using one or five examples show that our novel approach overcomes existing performance limitations, improving the state of the art both in few-shot semantic segmentation on the Graspnet-1B (+10.5% mIoU) and Ocid-grasp (+1.6% AP) datasets, and real-world few-shot grasp synthesis (+21.7% grasp accuracy). The project page is available at: https://leobarcellona.github.io/showandgrasp.github.io/

Show and Grasp: Few-shot Semantic Segmentation for Robot Grasping through Zero-shot Foundation Models

TL;DR

The paper addresses the limited generalization of few-shot semantic segmentation for robotic grasping by integrating zero-shot foundation models (Segment Anything Model and CutLER) with a high-performing Few-Shot Classifier (PMF). The method inverts the standard FSC role, using the zero-shot segmentator to generate candidate masks and the classifier to select the mask most similar to the support, with the resulting semantic heatmap guiding a grasp synthesis network that outputs a 4-DoF grasp. The approach achieves state-of-the-art results on Graspnet-1B (+10.5% mIoU) and OCID-grasp (+1.6% AP), plus notable real-world gains in grasp accuracy (+21.7%), all without any fine-tuning. Experiments on 47 real objects demonstrate robustness to appearance changes and instance-level vs. class-level generalization, while the paper also discusses limitations and future directions such as 7-DoF grasps and integration with Visual-Language Models for further gains.

Abstract

The ability of a robot to pick an object, known as robot grasping, is crucial for several applications, such as assembly or sorting. In such tasks, selecting the right target to pick is as essential as inferring a correct configuration of the gripper. A common solution to this problem relies on semantic segmentation models, which often show poor generalization to unseen objects and require considerable time and massive data to be trained. To reduce the need for large datasets, some grasping pipelines exploit few-shot semantic segmentation models, which are capable of recognizing new classes given a few examples. However, this often comes at the cost of limited performance and fine-tuning is required to be effective in robot grasping scenarios. In this work, we propose to overcome all these limitations by combining the impressive generalization capability reached by foundation models with a high-performing few-shot classifier, working as a score function to select the segmentation that is closer to the support set. The proposed model is designed to be embedded in a grasp synthesis pipeline. The extensive experiments using one or five examples show that our novel approach overcomes existing performance limitations, improving the state of the art both in few-shot semantic segmentation on the Graspnet-1B (+10.5% mIoU) and Ocid-grasp (+1.6% AP) datasets, and real-world few-shot grasp synthesis (+21.7% grasp accuracy). The project page is available at: https://leobarcellona.github.io/showandgrasp.github.io/
Paper Structure (22 sections, 4 equations, 7 figures, 4 tables)

This paper contains 22 sections, 4 equations, 7 figures, 4 tables.

Table of Contents

  1. INTRODUCTION
  2. RELATED WORKS
  3. Few-shot semantic segmentation for grasping
  4. Foundation models for zero-shot segmentation
  5. Few-shot classification
  6. DEFINITIONS
  7. METHOD
  8. Zero-shot segmentator
  9. Few-shot classifier
  10. Grasp synthesis network
  11. EXPERIMENTS
  12. Training
  13. Datasets and metrics
  14. Graspnet Dataset
  15. Ocid Dataset
  16. ...and 7 more sections

Figures (7)

  • Figure 1: Representation of the pipeline. The input is an RGB-D image and an example of the target object. To select the object, the segmentation is computed using a zero-shot segmentator and a few-shot classifier. The produced segmentation and the depth image are used to infer the grasp pose.
  • Figure 2: The few-shot semantic segmentation approach introduced in this work. The input image is given in input to a zero-shot segmentator (e.g., SAM). During the decoding, the features extracted from the grid sampled prompts are injected. The produced masks are used to filter the possible candidates. Each candidate is mapped into the feature space of a few-shot classifier together with the support features. The most similar features are used to extract the segmentation mask.
  • Figure 3: The grasp synthesis network. The semantic heatmap is extracted starting from the RGB and support image. Then, it is concatenated with the depth image to compose the input for three modules: Quality Head, Angle Head and Width Head. These modules produce three heatmaps encoding the grasp angle, width and position.
  • Figure 4: The robotic setup (left) and the set of objects (right).
  • Figure 5: The three real-world experimental setups. Experiment A (top) follows the previous test approach barcellona2023fsgFSGNET. Experiment B.1 (center) test the specificity i.e., the capability of recognizing a specific object inside a group of the same class. Experiment B.2 (bottom) investigates the generalizability i.e., the ability to recognize the target given a different object of the same class as support.
  • ...and 2 more figures