Causal Prompt Calibration Guided Segment Anything Model for Open-Vocabulary Multi-Entity Segmentation

TL;DR

This work tackles open-vocabulary multi-entity segmentation (OVMS) by diagnosing prompt bias as the root cause of SAM’s generalization failure. It formulates a causal perspective, defining causal prompts that encode only task-relevant factors and suppress confounders, and proves that cross-distribution consistency can yield such prompts. The authors introduce CPC-SAM, which embeds a lightweight Causal Prompt Learner (CaPL) into SAM and optimizes prompts and segmentation in a bi-level loop, enforcing causal multi-distribution consistency across tasks and entities. Through extensive experiments on nine datasets spanning general, medical, and complex scenes, CPC-SAM consistently outperforms state-of-the-art OVMS baselines, with strong performance under few-shot and OOD conditions. The approach offers a plug-and-play pathway to robust OVMS by automatically calibrating prompts without extra data, highlighting causal prompt learning as a practical route to improve VLM-based segmentation systems.

Abstract

Despite the strength of the Segment Anything Model (SAM), it struggles with generalization issues in open-vocabulary multi-entity segmentation (OVMS). Through empirical and causal analyses, we find that (i) the prompt bias is the primary cause of the generalization issues; (ii) this bias is closely tied to the task-irrelevant generating factors within the prompts, which act as confounders and affect generalization. To address the generalization issues, we aim to propose a method that can calibrate prompts to eliminate confounders for accurate OVMS. Building upon the causal analysis, we propose that the optimal prompt for OVMS should contain only task-relevant causal factors. We define it as the causal prompt, serving as the goal of calibration. Next, our theoretical analysis, grounded by causal multi-distribution consistency theory, proves that this prompt can be obtained by enforcing segmentation consistency and optimality. Inspired by this, we propose CPC-SAM, a Causal Prompt Calibration method for SAM to achieve accurate OVMS. It integrates a lightweight causal prompt learner (CaPL) into SAM to obtain causal prompts. Specifically, we first generate multiple prompts using random annotations to simulate diverse distributions and then reweight them via CaPL by enforcing causal multi-distribution consistency in both task and entity levels. To ensure obtaining causal prompts, CaPL is optimized by minimizing the cumulative segmentation loss across the reweighted prompts to achieve consistency and optimality. A bi-level optimization strategy alternates between optimizing CaPL and SAM, ensuring accurate OVMS. Extensive experiments validate its superiority.

Figures (22)

• Figure 1: Motivating experiments. (b) Existence of generalization issue, showing the dice score trends of training and unseen classes during training. (c) Existence of prompt bias, illustrating the dice score of SAM with prompts generated under different conditions. (d) Results on prompt bias across multiple entities, showing the impact of the same prompts on segmenting various entities.
• Figure 2: SCM of the prompt-tuning process. Solid and dashed circles are observable and unobservable variables, respectively.
• Figure 3: The framework of CPC-SAM. The yellow box represents CaPL and the green box and gray box are components for SAM.
• Figure 4: Performance comparison of complex scene and activity segmentation and visualization. (a) shows the performance gap ($\bigtriangleup$) of SAM and CPC-SAM vs. RITA on three datasets. (b) shows the OVMS visualization of CPC-SAM, with more results in Appendix \ref{['sec_app:experiment']}.
• Figure 5: Ablation study on prompt optimization methods.

Theorems & Definitions (3)

• Theorem 1
• Definition 1: Causal Prompt
• Definition 2