MTA-CLIP: Language-Guided Semantic Segmentation with Mask-Text Alignment

TL;DR

The paper tackles boundary ambiguities in CLIP-based semantic segmentation caused by low-resolution pixel-text alignment and global text embeddings. It introduces MTA-CLIP, a Mask-Text Alignment framework comprising a Mask-Text Decoder that jointly updates mask and text embeddings and a Mask-Text Prompt Learning module to capture diverse contexts, trained with a Mask-to-Text contrastive loss within the CLIP space ($\mathcal{L}_{sim}$). Key components include Text-Enhanced Mask Feature Learning, a cross-attention/self-attention mask decoder, and two negative-prompt strategies (MixNeg and SeparateNeg) to align each mask with the most relevant context text prompts. Empirical results on ADE20k and Cityscapes across multiple backbones show state-of-the-art performance, with an average improvement of $2.8\%$ on ADE20k and $1.3\%$ on Cityscapes over prior methods, demonstrating the effectiveness of mask-level language guidance.

Abstract

Recent approaches have shown that large-scale vision-language models such as CLIP can improve semantic segmentation performance. These methods typically aim for pixel-level vision-language alignment, but often rely on low resolution image features from CLIP, resulting in class ambiguities along boundaries. Moreover, the global scene representations in CLIP text embeddings do not directly correlate with the local and detailed pixel-level features, making meaningful alignment more difficult. To address these limitations, we introduce MTA-CLIP, a novel framework employing mask-level vision-language alignment. Specifically, we first propose Mask-Text Decoder that enhances the mask representations using rich textual data with the CLIP language model. Subsequently, it aligns mask representations with text embeddings using Mask-to-Text Contrastive Learning. Furthermore, we introduce MaskText Prompt Learning, utilizing multiple context-specific prompts for text embeddings to capture diverse class representations across masks. Overall, MTA-CLIP achieves state-of-the-art, surpassing prior works by an average of 2.8% and 1.3% on on standard benchmark datasets, ADE20k and Cityscapes, respectively.

Figures (4)

• Figure 1: (a): Pixel-Text Alignment relies on low-resolution image features for alignment and often encounters class boundary ambiguities. (b): t-sne visualisation for Pixel-Text vs. Mask-Text Alignment. Pixel features of "Wall" are misaligned with the text feature of "Person" for Pixel-Text Alignment. (c): Comparative results on the ADE20k, showcase the superior performance of Mask-Text Alignment in semantic segmentation tasks. on the ADE20k, showcase the superior performance of Mask-Text Alignment over Pixel-Text Alignment methods in semantic segmentation tasks.
• Figure 2: Comparison between Pixel-Text Alignment, Mask-Text Alignment (ours) and Mask-Text Matching segmentation architectures. Flames and snowflakes refer to learnable and frozen parameters, respectively.
• Figure 3: Overview of our framework. (a) MTA-CLIP introduces Mask-Text Alignment (\ref{['sub:mask-text']}) that adopts a novel set of input queries combining both the mask queries ($\mathbf{m}$) and text embeddings ($\hat{\mathbf{t}}$). These combined queries interact with each other within mask-text decoder layers, enhancing mask representations through the incorporation of rich textual data following Text-Enhanced Mask Feature Learning. Subsequently, class-level alignment is conducted between the mask and text representations following Mask-to-Text Contrastive Learning via a contrastive loss applied to the mask-to-text similarity scores. (b) Furthermore, distinct alignment of different context prompts with mask representations is achieved using Mask-Text Prompt Learning.
• Figure 4: Qualitative comparison. We compare our framework, MTA-CLIP's performance with Pixel-Text Alignment segmentation approach, DenseCLIP rao2022denseclip and Mask2Former cheng2022masked baseline with ResNet-50 backbone.