Disentangling Prompt Dependence to Evaluate Segmentation Reliability in Gynecological MRI

Abstract

Promptable segmentation models (e.g., the Segment Anything Models) enable generalizable, zero-shot segmentation across diverse domains. Although predictions are deterministic for a fixed image-prompt pair, the robustness of these models to variations in user prompts, referred to as prompt dependence, remains underexplored. In safety-critical workflows with substantial inter-user variability, interpretable and informative frameworks are needed to evaluate prompt dependence. In this work, we assess the reliability of promptable segmentation by analyzing and measuring its sensitivity to prompt variability. We introduce the first formulation of prompt dependence that explicitly disentangles prompt ambiguity (inter-user variability) from local sensitivity (interaction imprecision), offering an interpretable view of segmentation robustness. Experiments on two female pelvic MRI datasets for uterus and bladder segmentation reveal a strong negative correlation between both metrics and segmentation performance, highlighting the value of our framework for assessing robustness. The two metrics have low mutual correlation, supporting the disentangled design of our formulation, and provide meaningful indicators of prompt-related failure modes.

Figures (3)

• Figure 1: Overview of the prompt dependence evaluation framework. In prompt space (left), a conditional mixture model estimates the distribution of plausible prompts $P_{\text{valid}}(b\mid I)$. The dispersion of this distribution, quantified by the trace of its covariance, defines prompt ambiguity. By adding controlled perturbations around reference prompts and measuring their impact in mask space, we train a stability-margin predictor to estimate the minimum change needed to produce a meaningful discrepancy between masks, thereby providing a formulation of local sensitivity.
• Figure 2: Qualitative examples (top) and dataset-level summary statistics (bottom) of prompt dependence metrics, $U_{amb}$ and $\delta^\star$, for UT-EndoMRI and MOGaMBO. For each dataset, two representative slices are shown with the corresponding $U_{amb}$ and $\delta^\star$ values; the blue boxes highlight the ground-truth bounding box. The table reports the mean Dice (±SEM), the ranges of $U_{amb}$ and $\delta^\star$, and Pearson correlations on the test set. Stars $\star$ indicate significant correlations at $p<0.001$.
• Figure 3: (Top) Distribution of prompt ambiguity (x-axis) and local sensitivity (y-axis, encoded as $\log(\frac{1}{1-\delta^\star})$), color-coded by $1-Dice$. Dashed lines denote the median values used to define the four classes. (Bottom) Qualitative example of uncertainty maps produced by Jitter, DO, and CondMDN with the input image and ground truth mask on UT-EndoMRI, with averaged evaluation metrics of uncertainty maps.