Extract Free Dense Labels from CLIP

TL;DR

The paper addresses the difficulty of achieving dense, open-vocabulary segmentation without annotations or fine-tuning by leveraging CLIP’s visual-language representations.It introduces MaskCLIP, which preserves CLIP's alignment by extracting dense patch-level features from the image encoder and using text embeddings as classifiers, and MaskCLIP+ which uses pseudo labeling and self-training to train stronger segmentation models.MaskCLIP+ achieves state-of-the-art performance in transductive zero-shot segmentation across PASCAL VOC, PASCAL Context, and COCO Stuff, including marked gains for unseen classes, while maintaining strong performance on seen classes.The approach demonstrates that CLIP can serve as a robust, annotation-free source of supervision for dense prediction and is adaptable to diverse segmentation backbones.

Abstract

Contrastive Language-Image Pre-training (CLIP) has made a remarkable breakthrough in open-vocabulary zero-shot image recognition. Many recent studies leverage the pre-trained CLIP models for image-level classification and manipulation. In this paper, we wish examine the intrinsic potential of CLIP for pixel-level dense prediction, specifically in semantic segmentation. To this end, with minimal modification, we show that MaskCLIP yields compelling segmentation results on open concepts across various datasets in the absence of annotations and fine-tuning. By adding pseudo labeling and self-training, MaskCLIP+ surpasses SOTA transductive zero-shot semantic segmentation methods by large margins, e.g., mIoUs of unseen classes on PASCAL VOC/PASCAL Context/COCO Stuff are improved from 35.6/20.7/30.3 to 86.1/66.7/54.7. We also test the robustness of MaskCLIP under input corruption and evaluate its capability in discriminating fine-grained objects and novel concepts. Our finding suggests that MaskCLIP can serve as a new reliable source of supervision for dense prediction tasks to achieve annotation-free segmentation. Source code is available at https://github.com/chongzhou96/MaskCLIP.

Figures (7)

• Figure 1: Here we show the original image in (a), the segmentation result of MaskCLIP+ in (b), and the confidence maps of MaskCLIP and MaskCLIP+ for Batman in (c) and (d) respectively. Through the adaptation of CLIP, MaskCLIP can be directly used for segmentation of fine-grained and novel concepts (e.g., Batman and Joker) without any training operations and annotations. Combined with pseudo labeling and self-training, MaskCLIP+ further improves the segmentation result.
• Figure 2: Overview of MaskCLIP/MaskCLIP+.$\,\,\,$ Compared to the conventional fine-tuning method, the key to the success of MaskCLIP is keeping the pre-trained weights frozen and making minimal adaptation to preserve the visual-language association. Besides, to compensate for the weakness of using the CLIP image encoder for segmentation, which is designed for classification, MaskCLIP+ uses the outputs of MaskCLIP as pseudo labels and trains a more advanced segmentation network such as DeepLabv2 deeplabv2
• Figure 3: Qualitative results on PASCAL Context.$\,\,\,$ Here all results are obtained without any annotation. PD and KS refer to prompt denoising and key smoothing respectively. With PD, we can see some distraction classes are removed. KS is more aggressive. Its outputs are much less noisy but are dominated by a small number of classes. Finally, MaskCLIP+ yields the best results
• Figure 4: Qualitative results on Web images.$\,\,\,$ Here we show the segmentation results of MaskCLIP and MaskCLIP+ on various unseen classes, including fine-grained classes such as cars in different colors/imagery properties, celebrities, and animation characters. All results are obtained without any annotation
• Figure 5: More qualitative results on PASCAL Context.$\,\,\,$ Here all results are obtained without any annotation. PD and KS refer to prompt denoising and key smoothing respectively. Row 2, Col 4 shows a failure case of KS, where all the pixels in the image are labeled as the horse. Note that, PASCAL Context does not contain bear or teddy bear classes and MaskCLIP predicts the teddy bear pixels as bedclothes