DenseCLIP: Language-Guided Dense Prediction with Context-Aware Prompting

TL;DR

DenseCLIP tackles the challenge of transferring vision-language priors from CLIP to dense prediction tasks. It converts image-text matching into pixel-text matching to produce score maps that guide dense predictors, and uses context-aware prompting to refine language features with image context. The approach is model-agnostic and shows strong gains on semantic segmentation, object detection, and instance segmentation, while also working with backbones beyond CLIP. This work introduces a practical, plug-and-play framework that leverages large-scale vision-language supervision to improve dense vision tasks with notable gains and broad applicability.

Abstract

Recent progress has shown that large-scale pre-training using contrastive image-text pairs can be a promising alternative for high-quality visual representation learning from natural language supervision. Benefiting from a broader source of supervision, this new paradigm exhibits impressive transferability to downstream classification tasks and datasets. However, the problem of transferring the knowledge learned from image-text pairs to more complex dense prediction tasks has barely been visited. In this work, we present a new framework for dense prediction by implicitly and explicitly leveraging the pre-trained knowledge from CLIP. Specifically, we convert the original image-text matching problem in CLIP to a pixel-text matching problem and use the pixel-text score maps to guide the learning of dense prediction models. By further using the contextual information from the image to prompt the language model, we are able to facilitate our model to better exploit the pre-trained knowledge. Our method is model-agnostic, which can be applied to arbitrary dense prediction systems and various pre-trained visual backbones including both CLIP models and ImageNet pre-trained models. Extensive experiments demonstrate the superior performance of our methods on semantic segmentation, object detection, and instance segmentation tasks. Code is available at https://github.com/raoyongming/DenseCLIP

Figures (5)

• Figure 1: Comparisons of the conventional "pre-training + fine-tuning" paradigm and our proposed DenseCLIP. The pre-training + fine-tuning paradigm directly applies the image pre-trained model as the initialization of encoder. Differently, DenseCLIP transfers the knowledge learned with image-text contrastive learning to dense prediction models by introducing a new pixel-text matching task and further using the contextual information from images to prompt pre-trained language model.
• Figure 2: Results of different pre-training and fine-tuning strategies on the semantic segmentation task. We report the single-scale and multi-scale mIoU on ADE20K ade of different pre-trained ResNet-50 he2016deep models, including supervised ImageNet1K deng2009imagenet (IN1K) and ImageNet21K deng2009imagenetridnik2021imagenet (IN21K), self-supervised MoCoV2 mocov2 and DenseCL wang2021dense, and vision-language model CLIP. Equipped with DenseCLIP, we show that large-scale vision-language pre-training can substantially improve the dense prediction performance (+4.9%/+4.1%) over the commonly used ImageNet pre-training.
• Figure 3:
• Figure 4: Two different strategies of context-aware prompting. The pre-model prompting directly uses the image contexts to generate the desired text inputs, while post-model prompting refines the class embedding instead.
• Figure 5: Qualitative results on ADE20K. We visualize the segmentation results on ADE20K validation set of our DenseCLIP based on ResNet-101 and two baseline models.