Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

BREPS: Bounding-Box Robustness Evaluation of Promptable Segmentation

Andrey Moskalenko, Danil Kuznetsov, Irina Dudko, Anastasiia Iasakova, Nikita Boldyrev, Denis Shepelev, Andrei Spiridonov, Andrey Kuznetsov, Vlad Shakhuro

TL;DR

This paper investigates the robustness of promptable segmentation to real-world bounding-box prompts. It combines a large-scale real-user study (2,500 annotators, 25,000 bboxes) with a white-box adversarial framework (BREPS) that optimizes bounding-box prompts under a realism prior to test segmentation models across 10 datasets. The study reveals substantial inter-user variability and sensitivity to bbox perturbations, showing that tight bboxes often overestimate real-world performance, while small prompt changes can cause large IoU fluctuations. BREPS formalizes a scalable, realistic attack and demonstrates significant IoU degradation (≈$30\%$ on average under minimization) and limited gains under maximization, highlighting the need for robustness-aware training and evaluation in promptable segmentation. The work provides a dataset-driven realism prior (Gamma over CIoU-Loss) and a practical protocol, paving the way toward more reliable real-world deployment of bbox-guided segmentation systems.

Abstract

Promptable segmentation models such as SAM have established a powerful paradigm, enabling strong generalization to unseen objects and domains with minimal user input, including points, bounding boxes, and text prompts. Among these, bounding boxes stand out as particularly effective, often outperforming points while significantly reducing annotation costs. However, current training and evaluation protocols typically rely on synthetic prompts generated through simple heuristics, offering limited insight into real-world robustness. In this paper, we investigate the robustness of promptable segmentation models to natural variations in bounding box prompts. First, we conduct a controlled user study and collect thousands of real bounding box annotations. Our analysis reveals substantial variability in segmentation quality across users for the same model and instance, indicating that SAM-like models are highly sensitive to natural prompt noise. Then, since exhaustive testing of all possible user inputs is computationally prohibitive, we reformulate robustness evaluation as a white-box optimization problem over the bounding box prompt space. We introduce BREPS, a method for generating adversarial bounding boxes that minimize or maximize segmentation error while adhering to naturalness constraints. Finally, we benchmark state-of-the-art models across 10 datasets, spanning everyday scenes to medical imaging. Code - https://github.com/emb-ai/BREPS.

BREPS: Bounding-Box Robustness Evaluation of Promptable Segmentation

TL;DR

This paper investigates the robustness of promptable segmentation to real-world bounding-box prompts. It combines a large-scale real-user study (2,500 annotators, 25,000 bboxes) with a white-box adversarial framework (BREPS) that optimizes bounding-box prompts under a realism prior to test segmentation models across 10 datasets. The study reveals substantial inter-user variability and sensitivity to bbox perturbations, showing that tight bboxes often overestimate real-world performance, while small prompt changes can cause large IoU fluctuations. BREPS formalizes a scalable, realistic attack and demonstrates significant IoU degradation (≈ on average under minimization) and limited gains under maximization, highlighting the need for robustness-aware training and evaluation in promptable segmentation. The work provides a dataset-driven realism prior (Gamma over CIoU-Loss) and a practical protocol, paving the way toward more reliable real-world deployment of bbox-guided segmentation systems.

Abstract

Promptable segmentation models such as SAM have established a powerful paradigm, enabling strong generalization to unseen objects and domains with minimal user input, including points, bounding boxes, and text prompts. Among these, bounding boxes stand out as particularly effective, often outperforming points while significantly reducing annotation costs. However, current training and evaluation protocols typically rely on synthetic prompts generated through simple heuristics, offering limited insight into real-world robustness. In this paper, we investigate the robustness of promptable segmentation models to natural variations in bounding box prompts. First, we conduct a controlled user study and collect thousands of real bounding box annotations. Our analysis reveals substantial variability in segmentation quality across users for the same model and instance, indicating that SAM-like models are highly sensitive to natural prompt noise. Then, since exhaustive testing of all possible user inputs is computationally prohibitive, we reformulate robustness evaluation as a white-box optimization problem over the bounding box prompt space. We introduce BREPS, a method for generating adversarial bounding boxes that minimize or maximize segmentation error while adhering to naturalness constraints. Finally, we benchmark state-of-the-art models across 10 datasets, spanning everyday scenes to medical imaging. Code - https://github.com/emb-ai/BREPS.
Paper Structure (23 sections, 2 equations, 6 figures, 3 tables)

This paper contains 23 sections, 2 equations, 6 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Promptable Segmentation
  4. Adversarial Robustness
  5. Segmentation Prompts User Studies
  6. Real-Users Study
  7. Data Selection
  8. Crowdsourcing Setup
  9. Display Bias Elimination
  10. Real-Users Behavior Analysis
  11. Bounding-Box Quality Measuring
  12. Bounding-Box Quality Analysis
  13. Measuring the Bounding-Box Realism
  14. Models Robustness on Real-Users
  15. Bounding-Box Quality vs Model Performance
  16. ...and 8 more sections

Figures (6)

  • Figure 1: Top-left: image with ground-truth mask (green) and tight bbox (yellow). Bottom-left: real-users bboxes — desktop (cyan) and mobile (magenta). Top-right: IoU heatmap for SAM ViT-B; pixels shows IoU for a bbox anchored at that pixel and centered on the object. Bottom-right: IoU spread across 3 SOTA models for different users. We observed the large variability in IoU between individuals.
  • Figure 2: Top row: Berkeley, ADE20K, COCO, ACDC images with green mask and yellow tight bbox. Middle: SAM ViT-B IoU heatmaps—pixel color shows IoU (0 blue → 1 red) for a bbox cornered at that pixel and centred on the object. Bottom: user-drawn bboxes—desktop (cyan) vs mobile (magenta). Desktop prompts are tighter; user boxes vary and diverge from the tight bbox. Heatmaps reveal steep IoU drops from even 1-pixel shifts --- examples provided in Supplementary. Zoom for details.
  • Figure 3: IoU spread for 10 random instances (3 general + 1 medical datasets). Points: desktop (cyan), mobile (magenta), tight bbox (yellow). Columns sorted by mean IoU over 50 users. SAM and SAM2.1 show large inter-user variance, while tight bboxes consistently over-estimate quality.
  • Figure 4: CIoU-Loss and IoU density plots for real-users drawn vs. tight bboxes. Black dashed shows fitted Gamma PDF. Mobile prompts (magenta) skew to lower overlap --- we observe that on mobile devices, bboxes are more deviated from the tight bboxes than on desktop devices.
  • Figure 5: CIoU-Loss/IoU scatters for SAM ViT-B model; dataset labels show Spearman/Pearson correlations. We don't observe high correlations on 9 out of 10 datasets and associate the highest correlation of ACDC with the similarity and simplicity of instances in this dataset.
  • ...and 1 more figures