Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

SAIF: A Stability-Aware Inference Framework for Medical Image Segmentation with Segment Anything Model

Ke Wu, Shiqi Chen, Yiheng Zhong, Hengxian Liu, Yingxue Su, Yifang Wang, Junhao Jin, Guangyu Ren

Abstract

Segment Anything Model (SAM) enable scalable medical image segmentation but suffer from inference-time instability when deployed as a frozen backbone. In practice, bounding-box prompts often contain localization errors, and fixed threshold binarization introduces additional decision uncertainty. These factors jointly cause high prediction variance, especially near object boundaries, degrading reliability. We propose the Stability-Aware Inference Framework (SAIF), a training-free and plug-and-play inference framework that improves robustness by explicitly modeling prompt and threshold uncertainty. SAIF constructs a joint uncertainty space via structured box perturbations and threshold variations, evaluates each hypothesis using decision stability and boundary consistency, and introduces a stability-consistency score to filter unstable candidates and perform stability-weighted fusion in probability space. Experiments on Synapse, CVC-ClinicDB, Kvasir-SEG, and CVC-300 demonstrate that SAIF consistently improves segmentation accuracy and robustness, achieving state-of-the-art performance without retraining or architectural modification. Our anonymous code is released at https://anonymous.4open.science/r/SAIF.

SAIF: A Stability-Aware Inference Framework for Medical Image Segmentation with Segment Anything Model

Abstract

Segment Anything Model (SAM) enable scalable medical image segmentation but suffer from inference-time instability when deployed as a frozen backbone. In practice, bounding-box prompts often contain localization errors, and fixed threshold binarization introduces additional decision uncertainty. These factors jointly cause high prediction variance, especially near object boundaries, degrading reliability. We propose the Stability-Aware Inference Framework (SAIF), a training-free and plug-and-play inference framework that improves robustness by explicitly modeling prompt and threshold uncertainty. SAIF constructs a joint uncertainty space via structured box perturbations and threshold variations, evaluates each hypothesis using decision stability and boundary consistency, and introduces a stability-consistency score to filter unstable candidates and perform stability-weighted fusion in probability space. Experiments on Synapse, CVC-ClinicDB, Kvasir-SEG, and CVC-300 demonstrate that SAIF consistently improves segmentation accuracy and robustness, achieving state-of-the-art performance without retraining or architectural modification. Our anonymous code is released at https://anonymous.4open.science/r/SAIF.
Paper Structure (11 sections, 9 equations, 3 figures, 3 tables)

This paper contains 11 sections, 9 equations, 3 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Methodology
  3. Overview of the Proposed Framework
  4. Uncertainty Modeling
  5. Stability-Consistency Scoring
  6. Stability-Aware Fusion
  7. Experiment
  8. Experimental Setup
  9. Comparisons with State-of-the-art Methods
  10. Ablation Studies
  11. Conclusion

Figures (3)

  • Figure 1: Overview of medical adaptations of the Segment Anything Model. (a) Full retraining on large-scale medical data, (b) parameter-efficient fine-tuning, (c) structural modifications to the encoder and mask decoder, and (d) inference-time operating conditions affecting output reliability, addressed by approach.
  • Figure 2: This diagram shows the SAIF framework for stability-aware medical image segmentation. It starts with perturbing the original box prompt into multiple outer and inner candidates, which are processed through the model to generate probability maps. Stability-consistency scoring ranks candidates based on Soft IoU and percentiles. The top-$n$ candidates are selected and fused to produce the final segmentation output, using no additional network execution or training.
  • Figure 3: Hyper-parameter sensitivity and efficiency characterization on CVC-300 using MedSAM as base model. (a) Effect of inference budget on accuracy. (b) Sensitivity to $\tau$-grid resolution. (c) Sensitivity to top-n fusion size. (d) Deploy-time speed–accuracy trade-off.