Language-Driven Visual Consensus for Zero-Shot Semantic Segmentation

TL;DR

Equipped with a vision-language prompting strategy, this approach significantly boosts the generalization capacity of segmentation models for unseen classes and introduces route attention into the transformer decoder to find visual consensus, thereby enhancing semantic consistency within the same object.

Abstract

The pre-trained vision-language model, exemplified by CLIP, advances zero-shot semantic segmentation by aligning visual features with class embeddings through a transformer decoder to generate semantic masks. Despite its effectiveness, prevailing methods within this paradigm encounter challenges, including overfitting on seen classes and small fragmentation in masks. To mitigate these issues, we propose a Language-Driven Visual Consensus (LDVC) approach, fostering improved alignment of semantic and visual information.Specifically, we leverage class embeddings as anchors due to their discrete and abstract nature, steering vision features toward class embeddings. Moreover, to circumvent noisy alignments from the vision part due to its redundant nature, we introduce route attention into self-attention for finding visual consensus, thereby enhancing semantic consistency within the same object. Equipped with a vision-language prompting strategy, our approach significantly boosts the generalization capacity of segmentation models for unseen classes. Experimental results underscore the effectiveness of our approach, showcasing mIoU gains of 4.5 on the PASCAL VOC 2012 and 3.6 on the COCO-Stuff 164k for unseen classes compared with the state-of-the-art methods.

Figures (5)

• Figure 1: We visualize the cross attention maps between image features and class embeddings from last transformer decoder block in both our approach and the SOTA method, i.e., ZegCLIP zhou2022zegclip. It shows that our approach establishes a clearer, more explicit alignment between dense image features and class embeddings.
• Figure 2: Illustration of difference between Our Decoder and ZegCLIP's Decoder zhou2022zegclip. We depict the change of the visual and semantic spaces before and after the decoder. Our approach initially contracts the region of the visual space, maximizing its similarity to the class embeddings. In contrast, ZegCLIP zhou2022zegclip pushes class embeddings towards noisy and redundant visual cues, resulting in significant semantic drift.
• Figure 3: The architecture of the proposed Language Driven Visual Consensus (LDVC) approach. We freeze the parameters of CLIP and insert deep learnable visual-language prompts into encoders. Then a text adapter fuses class embeddings $T$ and global image feature $V_g$. At last, the local-consensus transformer decoder progressively updates the dense visual features $V_l$ under the guidance of updated class embeddings $T_{new}$ to get the final segmentation logits $S$.
• Figure 4: Visualization of segmentation prediction under inductive settings on COCO-Stuff 164k. We compare the segmentation results between our approach and the SOTA method. The categories with color marked are unseen classes and others are seen classes.
• Figure 5: Visualization of cross attention between image features and class embeddings in our decoder with LCSA or without LCSA.