Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

ProMi: An Efficient Prototype-Mixture Baseline for Few-Shot Segmentation with Bounding-Box Annotations

Florent Chiaroni, Ali Ayub, Ola Ahmad

TL;DR

ProMi tackles the challenge of few-shot binary segmentation under bounding-box supervision by introducing a training-free prototype-mixture classifier that models the background as a distributional mixture and converts bounding boxes into noisy patch-level labels in latent space. The method alternates between feature extraction, patch-label conversion, and iterative prototype refinement, adding background prototypes up to a maximum and refining the foreground prototype to counteract label noise. Across standard benchmarks and with a powerful foundation model (Dinov2), ProMi achieves state-of-the-art mean-IoU while maintaining efficiency, and demonstrates practical usefulness on real-world robotic datasets (underwater, aerial, and urban). The work reduces annotation burden and enables rapid adaptation to new objects and environments in robotics, with future potential for multi-class segmentation and automated annotation strategies.

Abstract

In robotics applications, few-shot segmentation is crucial because it allows robots to perform complex tasks with minimal training data, facilitating their adaptation to diverse, real-world environments. However, pixel-level annotations of even small amount of images is highly time-consuming and costly. In this paper, we present a novel few-shot binary segmentation method based on bounding-box annotations instead of pixel-level labels. We introduce, ProMi, an efficient prototype-mixture-based method that treats the background class as a mixture of distributions. Our approach is simple, training-free, and effective, accommodating coarse annotations with ease. Compared to existing baselines, ProMi achieves the best results across different datasets with significant gains, demonstrating its effectiveness. Furthermore, we present qualitative experiments tailored to real-world mobile robot tasks, demonstrating the applicability of our approach in such scenarios. Our code: https://github.com/ThalesGroup/promi.

ProMi: An Efficient Prototype-Mixture Baseline for Few-Shot Segmentation with Bounding-Box Annotations

TL;DR

ProMi tackles the challenge of few-shot binary segmentation under bounding-box supervision by introducing a training-free prototype-mixture classifier that models the background as a distributional mixture and converts bounding boxes into noisy patch-level labels in latent space. The method alternates between feature extraction, patch-label conversion, and iterative prototype refinement, adding background prototypes up to a maximum and refining the foreground prototype to counteract label noise. Across standard benchmarks and with a powerful foundation model (Dinov2), ProMi achieves state-of-the-art mean-IoU while maintaining efficiency, and demonstrates practical usefulness on real-world robotic datasets (underwater, aerial, and urban). The work reduces annotation burden and enables rapid adaptation to new objects and environments in robotics, with future potential for multi-class segmentation and automated annotation strategies.

Abstract

In robotics applications, few-shot segmentation is crucial because it allows robots to perform complex tasks with minimal training data, facilitating their adaptation to diverse, real-world environments. However, pixel-level annotations of even small amount of images is highly time-consuming and costly. In this paper, we present a novel few-shot binary segmentation method based on bounding-box annotations instead of pixel-level labels. We introduce, ProMi, an efficient prototype-mixture-based method that treats the background class as a mixture of distributions. Our approach is simple, training-free, and effective, accommodating coarse annotations with ease. Compared to existing baselines, ProMi achieves the best results across different datasets with significant gains, demonstrating its effectiveness. Furthermore, we present qualitative experiments tailored to real-world mobile robot tasks, demonstrating the applicability of our approach in such scenarios. Our code: https://github.com/ThalesGroup/promi.
Paper Structure (20 sections, 4 equations, 3 figures, 4 tables)

This paper contains 20 sections, 4 equations, 3 figures, 4 tables.

Table of Contents

  1. INTRODUCTION
  2. RELATED WORK
  3. Few-Shot Learning
  4. Few-Shot Segmentation
  5. Weakly-Supervised Few-Shot Segmentation
  6. PROBLEM FORMULATION
  7. Target challenge: Few-Shot Binary Segmentation with Bounding-box Annotations
  8. Motivation for Our Prototype-Based Strategy
  9. PROPOSED APPROACH
  10. Feature Extraction
  11. Annotation Conversion: From Bounding-Box to Latent Space-Ready Patch-Level Labels.
  12. ProMi: Prototype mixture-based learning in latent space
  13. Inference on the Query set
  14. EXPERIMENTS
  15. Experiments Overview
  16. ...and 5 more sections

Figures (3)

  • Figure 1: Qualitative results on mobile robot datasets. Our approach was applied in the 1-shot setting to segment the "underwater robot" class on SUIM islam2020semantic (first row) and the "car" class on Cityscapes Cordts2016Cityscapes (second row). The first column shows 1-shot annotations, followed by three corresponding predictions.
  • Figure 2: Qualitative result on the high-resolution UAVid dataset. Our method was trained on a small annotated window, outlined in white in the bottom left of the image, and then used to infer segmentation predictions on the remaining areas.
  • Figure 3: ProMi mean-IoU scores as a function of the number of background prototypes, on PASCAL VOC 2012 using the Dinov2 model (ViT-B/14 backbone).