SAM-MI: A Mask-Injected Framework for Enhancing Open-Vocabulary Semantic Segmentation with SAM

TL;DR

Open-vocabulary semantic segmentation seeks universal pixel-level recognition, but direct integration of SAM often yields over-segmentation and rigid mask-label coupling. SAM-MI mitigates this by a mask-injected pipeline comprising a Text-guided Sparse Point Prompter to accelerate mask generation, Shallow Mask Aggregation to merge redundant masks, and Decoupled Mask Injection to guide cost maps with SAM-derived context in low- and high-frequency channels. The approach yields strong gains across multiple OVSS benchmarks, notably a 16.7% relative mIoU improvement on MESS and a 1.6× speedup, while maintaining robustness to low-quality masks. These results demonstrate a practical pathway to couple SAM with OVSS for improved accuracy and efficiency, with potential extensions to broader dense-prediction tasks.

Abstract

Open-vocabulary semantic segmentation (OVSS) aims to segment and recognize objects universally. Trained on extensive high-quality segmentation data, the segment anything model (SAM) has demonstrated remarkable universal segmentation capabilities, offering valuable support for OVSS. Although previous methods have made progress in leveraging SAM for OVSS, there are still some challenges: (1) SAM's tendency to over-segment and (2) hard combinations between fixed masks and labels. This paper introduces a novel mask-injected framework, SAM-MI, which effectively integrates SAM with OVSS models to address these challenges. Initially, SAM-MI employs a Text-guided Sparse Point Prompter to sample sparse prompts for SAM instead of previous dense grid-like prompts, thus significantly accelerating the mask generation process. The framework then introduces Shallow Mask Aggregation (SMAgg) to merge partial masks to mitigate the SAM's over-segmentation issue. Finally, Decoupled Mask Injection (DMI) incorporates SAM-generated masks for guidance at low-frequency and high-frequency separately, rather than directly combining them with labels. Extensive experiments on multiple benchmarks validate the superiority of SAM-MI. Notably, the proposed method achieves a 16.7% relative improvement in mIoU over Grounded-SAM on the MESS benchmark, along with a 1.6$\times$ speedup. We hope SAM-MI can serve as an alternative methodology to effectively equip the OVSS model with SAM.

Figures (10)

• Figure 1: Illustration of the challenges addressed by SAM-MI. Previous OVSS methods integrating SAM suffer from the following issues: (i) over-segmentation, SAM may segment numerous out-of-interest patches caused by the edges; (ii) hard combination, previous mask-driven and prompt-driven methods combine fixed masks and labels directly, ignoring the incorrect or low-quality masks. Our proposed mask-injected framework incorporates Shallow Mask Aggregation (SMAgg) to alleviate the over-segmentation issue, and Decoupled Mask Injection (DMI) to avoid the hard combination. Additionally, we accelerate the SAM's masks generation process via learnable sparse point prompts.
• Figure 2: The overall architecture of SAM-MI. It begins by utilizing the visual and text encoder from CLIP to produce dense pixel-text cost maps $S$. Subsequently, the Text-guided Sparse Point Prompter is employed to create sparse point prompts for SAM to generate mask proposals $M^{sam}$, which are then partially aggregated by the Shallow Mask Aggregation. Finally, the Decoupled Mask Injection is adopted to inject contextual information from $M^{sam}$ into $S$ in a low-frequency and high-frequency decoupled manner.
• Figure 3: Illustration of Text-guided Sparse Point Prompter. (a) Compared with existing dense grid-like point prompts, our Text-guided sparse point prompts achieve comparable masks generated by SAM with much fewer point prompts. (b) The points sampled by TSPP follow two assumptions. (c) The architecture of TSPP consists of two linear projection heads.
• Figure 4: Design of Decoupled Mask Injection. (a) The low-frequency injection models intra-mask context by mask-pooling and inter-mask context by cross attention in low-resolution feature map $\mathcal{F}_{l}^v$. (b) The high-frequency injection extracts and injects high-frequency representations from the high-resolution feature maps $\mathcal{F}_{h1}^v$ and $F_{h2}^v$.
• Figure 5: Qualitative comparison with SED xie2024sed and CAT-Seg cho2024cat on ADE20K-150 validation set. SAM-MI is capable of producing more accurate masks.