RegionCLIP: Region-based Language-Image Pretraining

TL;DR

RegionCLIP tackles the lack of region-grounded semantics in CLIP by introducing region-level vision-language pretraining. It bootstraps region descriptions from a large concept pool, aligns region crops to these descriptions via a teacher CLIP model, and optimizes with a combination of contrastive and distillation losses to produce rich regional representations. The pretrained region encoder transfers to open-vocabulary and zero-shot object detection, delivering state-of-the-art results on COCO and LVIS and enabling flexible region-based grounding without manual region annotations. The approach demonstrates the viability and practicality of region-focused vision-language pretraining for fine-grained detection tasks and offers a scalable path to grounding textual concepts in image regions.

Abstract

Contrastive language-image pretraining (CLIP) using image-text pairs has achieved impressive results on image classification in both zero-shot and transfer learning settings. However, we show that directly applying such models to recognize image regions for object detection leads to poor performance due to a domain shift: CLIP was trained to match an image as a whole to a text description, without capturing the fine-grained alignment between image regions and text spans. To mitigate this issue, we propose a new method called RegionCLIP that significantly extends CLIP to learn region-level visual representations, thus enabling fine-grained alignment between image regions and textual concepts. Our method leverages a CLIP model to match image regions with template captions and then pretrains our model to align these region-text pairs in the feature space. When transferring our pretrained model to the open-vocabulary object detection tasks, our method significantly outperforms the state of the art by 3.8 AP50 and 2.2 AP for novel categories on COCO and LVIS datasets, respectively. Moreoever, the learned region representations support zero-shot inference for object detection, showing promising results on both COCO and LVIS datasets. Our code is available at https://github.com/microsoft/RegionCLIP.

Figures (4)

• Figure 1: (a). A pretrained CLIP model radford2021learning failed to capture localization quality. (b). A major drop on accuracy when using the same pretrained CLIP to classify image regions. (c). Our key idea is learning to match image regions and their text descriptions.
• Figure 2: Method overview. We propose to learn visual representation for image regions via vision-language pretraining. Panel 1: With contrastive learning, CLIP is able to match images and their descriptions. Panel 2: Initialized by pretrained CLIP, our visual encoder learns visual region representation from the created region-text pairs. Specifically, as shown in the bottom row, we first create texts by filling the prompts with object concepts which are parsed from image descriptions, then use pretrained CLIP to align these texts and image regions proposed by RPN. Panel 3: When human annotation for image regions is available, we transfer our visual encoder for object detection.
• Figure 3: Visualization of zero-shot inference on COCO dataset with ground-truth boxes. Without finetuning, the pretrained models (top: CLIP, bottom: Ours) are directly used to recognize image regions into 65 categories in COCO. (Image IDs: 9448, 9483, 7386, 4795)
• Figure 4: Visualization of zero-shot inference on COCO dataset with ground-truth boxes. Without finetuning, the pretrained models are asked to predict 1203 categories from LVIS dataset. We show the top-3 predicted categories from our pretrained model and pretrained CLIP model. (Image IDs: 776, 13597, 17029)