DPSeg: Dual-Prompt Cost Volume Learning for Open-Vocabulary Semantic Segmentation

TL;DR

The paper tackles open-vocabulary semantic segmentation by addressing the domain gap between image and text embeddings and by leveraging shallow, fine-grained features. It introduces DPSeg, a dual-prompt cost volume framework that combines visual and text prompts to generate a rich cost volume, guided by a cost volume-decoder, and enhanced by a semantic-guided prompt refinement in a two-pass inference. Key contributions include a visual-prompt–driven enhancement of intra-modal alignment, a multi-scale cost-volume fusion strategy, and an instance-aware refinement mechanism that improves segmentation accuracy for small or ambiguous objects. Experimental results across five OVSS benchmarks demonstrate state-of-the-art performance and robustness to prompt quality, highlighting DPSeg’s practical impact for robust open-category segmentation in real-world scenes.

Abstract

Open-vocabulary semantic segmentation aims to segment images into distinct semantic regions for both seen and unseen categories at the pixel level. Current methods utilize text embeddings from pre-trained vision-language models like CLIP but struggle with the inherent domain gap between image and text embeddings, even after extensive alignment during training. Additionally, relying solely on deep text-aligned features limits shallow-level feature guidance, which is crucial for detecting small objects and fine details, ultimately reducing segmentation accuracy. To address these limitations, we propose a dual prompting framework, DPSeg, for this task. Our approach combines dual-prompt cost volume generation, a cost volume-guided decoder, and a semantic-guided prompt refinement strategy that leverages our dual prompting scheme to mitigate alignment issues in visual prompt generation. By incorporating visual embeddings from a visual prompt encoder, our approach reduces the domain gap between text and image embeddings while providing multi-level guidance through shallow features. Extensive experiments demonstrate that our method significantly outperforms existing state-of-the-art approaches on multiple public datasets.

Figures (8)

• Figure 1: T-SNE visualization of embeddings in CLIP feature space, showing text prompt, visual prompt, target image, and our dual-prompt for a street scene image. Numbers indicate distance to image feature, where our dual-prompt approach achieves closest proximity (0.18).
• Figure 2: Visualization of cosine similarities comparing image embeddings $\mathbf{E}$ with text prompt embeddings $\mathbf{T}$ (blue dots) and visual prompt embeddings $\mathbf{V}$ (green crosses) across sampled categories.
• Figure 3: Visualization of cost volume: (a) image with 'mirror' and 'bed' segments; (b) cost volume with text prompts; (c) cost volume with visual prompts; (d) aggregated cost volume. The top row represents the unseen class 'mirror,' and the bottom row represents the seen class 'bed'.
• Figure 4: Architecture of our DPSeg network. We begin by generating visual prompts based on text prompt templates, which are then combined with text prompts to create the dual-prompt cost volume. Subsequently, we incorporate multi-scale image features and corresponding visual prompt embeddings into the cost volume-guided decoder (CVGD) in a progressive manner.
• Figure 5: Structure of Semantic-Guided Prompt Refinement.