Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

On The Robustness of Foundational 3D Medical Image Segmentation Models Against Imprecise Visual Prompts

Soumitri Chattopadhyay, Basar Demir, Marc Niethammer

TL;DR

This work analyzes the robustness of promptable 3D medical segmentation against imprecise dense visual prompts using two state-of-the-art models, nnInteractive and SAM-Med3D, on the BTCV abdominal multi-organ dataset. It introduces controlled perturbations—morphological dilations/erosions, boundary-preserving erosion, and spatial translations—to assess reliance on shape and boundary priors. Findings show that prompts preserving boundary information markedly improve robustness, while standard erosions and dilations degrade performance, especially for smaller or irregular organs; SAM-Med3D generally offers greater stability to perturbations and translations than nnInteractive. The results highlight the importance of shape priors in prompt-driven 3D segmentation and motivate training strategies to enhance robustness for clinical deployment.

Abstract

While 3D foundational models have shown promise for promptable segmentation of medical volumes, their robustness to imprecise prompts remains under-explored. In this work, we aim to address this gap by systematically studying the effect of various controlled perturbations of dense visual prompts, that closely mimic real-world imprecision. By conducting experiments with two recent foundational models on a multi-organ abdominal segmentation task, we reveal several facets of promptable medical segmentation, especially pertaining to reliance on visual shape and spatial cues, and the extent of resilience of models towards certain perturbations. Codes are available at: https://github.com/ucsdbiag/Prompt-Robustness-MedSegFMs

On The Robustness of Foundational 3D Medical Image Segmentation Models Against Imprecise Visual Prompts

TL;DR

This work analyzes the robustness of promptable 3D medical segmentation against imprecise dense visual prompts using two state-of-the-art models, nnInteractive and SAM-Med3D, on the BTCV abdominal multi-organ dataset. It introduces controlled perturbations—morphological dilations/erosions, boundary-preserving erosion, and spatial translations—to assess reliance on shape and boundary priors. Findings show that prompts preserving boundary information markedly improve robustness, while standard erosions and dilations degrade performance, especially for smaller or irregular organs; SAM-Med3D generally offers greater stability to perturbations and translations than nnInteractive. The results highlight the importance of shape priors in prompt-driven 3D segmentation and motivate training strategies to enhance robustness for clinical deployment.

Abstract

While 3D foundational models have shown promise for promptable segmentation of medical volumes, their robustness to imprecise prompts remains under-explored. In this work, we aim to address this gap by systematically studying the effect of various controlled perturbations of dense visual prompts, that closely mimic real-world imprecision. By conducting experiments with two recent foundational models on a multi-organ abdominal segmentation task, we reveal several facets of promptable medical segmentation, especially pertaining to reliance on visual shape and spatial cues, and the extent of resilience of models towards certain perturbations. Codes are available at: https://github.com/ucsdbiag/Prompt-Robustness-MedSegFMs
Paper Structure (7 sections, 3 equations, 6 figures)

This paper contains 7 sections, 3 equations, 6 figures.

Table of Contents

  1. Introduction
  2. Materials and Methods
  3. Experimental Setup
  4. Findings and Analysis
  5. Conclusion
  6. Acknowledgements
  7. Compliance with Ethical Standards

Figures (6)

  • Figure 1: Mean trends aggregated over all organs for different prompt perturbations. The respective color bands denote standard deviations across all samples. Models show greater resilience to dilated prompts as compared to eroded ones, as well as to prompts containing bounding shape information.
  • Figure 2: Qualitative results of nnInteractive for different prompt perturbations. Inset shows zoomed-in regions containing the organs, their perturbed prompts, predictions and the gold standard. As can be seen, the model over- and under-segments for dilated and eroded prompts, while it is more reliable when the prompt has bounding voxels ("Cavity").
  • Figure 3: Qualitative results of SAM-Med3D for different prompt perturbations. Each row depicts a different organ.
  • Figure 4: Segmentation trends across varying strengths of prompt perturbations (radius for dilation/erosion, boundary thickness for boundary-preserved erosion) for different organs. The respective color bands depict the standard deviation across all samples.
  • Figure 5: Organ-wise performance for spatially shifted prompts, for SAM-Med3D (top) and nnInteractive (bottom).
  • ...and 1 more figures