Open-Vocabulary Semantic Segmentation with Mask-adapted CLIP

TL;DR

This work tackles open-vocabulary semantic segmentation by identifying CLIP's poor performance on masked region inputs as a bottleneck in two-stage methods. It proposes adapting CLIP to masked images through diverse, caption-derived mask-category pairs and a novel mask prompt tuning technique that injects learnable prompts for masked patches without altering CLIP weights. Empirical results show significant gains over prior open-vocabulary approaches across ADE20K, Pascal Context, and PASCAL VOC, with the largest model achieving state-of-the-art zero-shot performance and even rivaling some supervised specialists without dataset-specific adaptation. The findings highlight the potential of open-vocabulary generalist models to generalize to unseen categories in a practical, multi-task setting.

Abstract

Open-vocabulary semantic segmentation aims to segment an image into semantic regions according to text descriptions, which may not have been seen during training. Recent two-stage methods first generate class-agnostic mask proposals and then leverage pre-trained vision-language models, e.g., CLIP, to classify masked regions. We identify the performance bottleneck of this paradigm to be the pre-trained CLIP model, since it does not perform well on masked images. To address this, we propose to finetune CLIP on a collection of masked image regions and their corresponding text descriptions. We collect training data by mining an existing image-caption dataset (e.g., COCO Captions), using CLIP to match masked image regions to nouns in the image captions. Compared with the more precise and manually annotated segmentation labels with fixed classes (e.g., COCO-Stuff), we find our noisy but diverse dataset can better retain CLIP's generalization ability. Along with finetuning the entire model, we utilize the "blank" areas in masked images using a method we dub mask prompt tuning. Experiments demonstrate mask prompt tuning brings significant improvement without modifying any weights of CLIP, and it can further improve a fully finetuned model. In particular, when trained on COCO and evaluated on ADE20K-150, our best model achieves 29.6% mIoU, which is +8.5% higher than the previous state-of-the-art. For the first time, open-vocabulary generalist models match the performance of supervised specialist models in 2017 without dataset-specific adaptations.

Figures (8)

• Figure 1: (a) CLIP is pre-trained with natural images with little data augmentation. (b) Two-stage open-vocabulary semantic segmentation approaches first generate class-agnostic mask proposals and then leverage pre-trained CLIP to do open-vocabulary classification. The input of the CLIP model is cropped masked images, which have huge domain gap from the natural images. (c) Our analysis reveals that pre-trained CLIP does not work well on masked images.
• Figure 2: Two-stage approaches consist of one segmentation model, e.g., MaskFormer, and one CLIP model. Firstly, the modified MaskFormer is trained with CLIP’s text embeddings so as to perform open-vocabulary segmentation. (Section \ref{['sec:two_stage_approach']}). We then use the pre-trained segmentation model to generate class-agnostic proposals and align proposals with extracted nouns from corresponding captions (Section \ref{['sec:collecting_diverse_pairs']}). After collecting diverse mask-category pairs, we finetune CLIP with the proposed mask prompt tuning (Section \ref{['sec:method_mask_prompt_learning']}).
• Figure 3: For the given image-cation pair, only "apple" and "orange" are categories in COCO. By extracting nouns from captions, we can also get a novel "teapot" category.
• Figure 4: The proposed mask prompt tuning can adapt CLIP to masked images without changing its weights. We replace the zero tokens from masked patches with learnable mask prompts.
• Figure 5: Open-vocabulary segmentation with user-defined queries. Our model accurately segments unseen categories, such as the Saturn V rocket, Oculus headset, and Golden gate bridge.