Effective SAM Combination for Open-Vocabulary Semantic Segmentation

TL;DR

This work tackles open-vocabulary semantic segmentation by bridging CLIP-based image-text correlation with SAM’s promptable segmentation in a one-stage framework. ESC-Net introduces correlation-derived pseudo prompts that feed SAM blocks, coupled with a Vision-Language Fusion module and a lightweight decoder, to produce dense, class-agnostic masks without a separate mask proposal stage. It achieves state-of-the-art results on ADE20K, PASCAL-VOC, and PASCAL-Context, with notable gains on high-class-count splits, and includes comprehensive ablations on SAM usage and prompt strategies. The approach offers a practical, efficient path for open-vocabulary segmentation, enabling robust performance with reduced inference cost and memory overhead compared to prior two-stage and one-stage methods.

Abstract

Open-vocabulary semantic segmentation aims to assign pixel-level labels to images across an unlimited range of classes. Traditional methods address this by sequentially connecting a powerful mask proposal generator, such as the Segment Anything Model (SAM), with a pre-trained vision-language model like CLIP. But these two-stage approaches often suffer from high computational costs, memory inefficiencies. In this paper, we propose ESC-Net, a novel one-stage open-vocabulary segmentation model that leverages the SAM decoder blocks for class-agnostic segmentation within an efficient inference framework. By embedding pseudo prompts generated from image-text correlations into SAM's promptable segmentation framework, ESC-Net achieves refined spatial aggregation for accurate mask predictions. ESC-Net achieves superior performance on standard benchmarks, including ADE20K, PASCAL-VOC, and PASCAL-Context, outperforming prior methods in both efficiency and accuracy. Comprehensive ablation studies further demonstrate its robustness across challenging conditions.

Figures (6)

• Figure 1: (a) A model structure that generates proposal masks using a mask generation model. (b) A model structure that refines the correlation between image and text. (c) The structure of the proposed ESC-Net. Our ESC-Net efficiently models the relationship between images and text by combining a pre-trained SAM block with pseudo prompts instead of an inefficient mask generation model. This approach enables much denser mask prediction compared to conventional correlation-based methods.
• Figure 2: The proposed ESC-Net consists of the CLIP vision and language encoders, $N$ consecutive ESCBlocks, and a decoder. Each ESC-Block generates a pseudo prompt from the image-text correlation map and uses it as input to the SAM block. The SAM block aggregates the CLIP image features. The VLF block models the image-text correlation using image features and text features, refining the correlation map through this process.
• Figure 3: The process of the proposed Pseudo Prompt Generator (PPG). PPG aims to generate class-specific pseudo prompts from image-text correlation maps. For efficiency, all processes are computed in batch-wise parallelization over all classes.
• Figure 4: The structure of the proposed Vision-Language Fusion (VLF) block. VLF sequentially applies image and text guidance to the correlation map to refine it.
• Figure 5: Qualitative comparison of CAT-Seg and our ESC-Net across various datasets. Our model is capable of generating more accurate and robust masks compared to existing correlation-based state-of-the-art method.