DiSa: Saliency-Aware Foreground-Background Disentangled Framework for Open-Vocabulary Semantic Segmentation

TL;DR

DiSa tackles the bias of vision-language priors toward salient foregrounds in open-vocabulary semantic segmentation by introducing a Saliency-aware Disentanglement Module (SDM) that splits per-class visual embeddings into foreground $C_f$ and background $C_b$ using saliency maps derived from cross-attention, and a Hierarchical Refinement Module (HRM) that refines these representations at the pixel, category, and semantic levels. The approach leverages an auxiliary Image-Text Matching loss to sharpen saliency via GradCAM-style gradients, while a gating-based Foreground/Background Aggregation fuses refined features into unified correlation maps for mask prediction. Across six benchmarks, DiSa demonstrates consistent improvements over state-of-the-art methods, with notable gains on background-inclusive settings and improved boundary localization, while maintaining a lightweight inference path. The work advances open-vocabulary segmentation by explicitly modeling context-dependent foreground and background semantics and by refining their representations through multi-level spatial and channelwise refinements, enabling better generalization to novel categories and complex scenes.

Abstract

Open-vocabulary semantic segmentation aims to assign labels to every pixel in an image based on text labels. Existing approaches typically utilize vision-language models (VLMs), such as CLIP, for dense prediction. However, VLMs, pre-trained on image-text pairs, are biased toward salient, object-centric regions and exhibit two critical limitations when adapted to segmentation: (i) Foreground Bias, which tends to ignore background regions, and (ii) Limited Spatial Localization, resulting in blurred object boundaries. To address these limitations, we introduce DiSa, a novel saliency-aware foreground-background disentangled framework. By explicitly incorporating saliency cues in our designed Saliency-aware Disentanglement Module (SDM), DiSa separately models foreground and background ensemble features in a divide-and-conquer manner. Additionally, we propose a Hierarchical Refinement Module (HRM) that leverages pixel-wise spatial contexts and enables channel-wise feature refinement through multi-level updates. Extensive experiments on six benchmarks demonstrate that DiSa consistently outperforms state-of-the-art methods.

Figures (11)

• Figure 1: Visualization of correlation maps. VLMs face Foreground Bias and Limited Spatial Localization challenges. Our proposed DiSa effectively alleviates these challenges. Row 1 indicates class “building”, and Row 2 indicates class “animal”.
• Figure 2: Overview of the DiSa Framework. DiSa consists of a Saliency-aware Disentanglement Module (SDM) and a Hierarchical Refinement Module (HRM), followed by an upsampling decoder.
• Figure 3: Illustration of Hierarchical Refinement Module (HRM).
• Figure 4: Qualitative results compared to CAT-Seg. DiSa produces more accurate predictions of small objects and visually-similar regions compared to existing SOTA methods. More qualitative results are in the appendix.
• Figure 5: Qualitative comparison of saliency and predictions.