Region-centric Image-Language Pretraining for Open-Vocabulary Detection

TL;DR

This work tackles open-vocabulary detection by matching region-level recognition with large-scale image-text pretraining. It introduces DITO, which integrates detector heads into CLIP pretraining (region-centric pretraining) and adds Shifted-Window Learning to mitigate window biases during high-resolution adaptation. Key contributions include a practical region-centric pretraining scheme without pseudo-labels, a novel SWL technique, and a distillation strategy to preserve open-vocabulary knowledge. Empirically, DITO achieves state-of-the-art $AP_r$ on LVIS (37.6 with LAION-2B and 40.9 with DataComp-1B) and competitive results on COCO, plus strong transfer performance, demonstrating enhanced localization and generalization for open-vocabulary detectors.

Abstract

We present a new open-vocabulary detection approach based on region-centric image-language pretraining to bridge the gap between image-level pretraining and open-vocabulary object detection. At the pretraining phase, we incorporate the detector architecture on top of the classification backbone, which better serves the region-level recognition needs of detection by enabling the detector heads to learn from large-scale image-text pairs. Using only standard contrastive loss and no pseudo-labeling, our approach is a simple yet effective extension of the contrastive learning method to learn emergent object-semantic cues. In addition, we propose a shifted-window learning approach upon window attention to make the backbone representation more robust, translation-invariant, and less biased by the window pattern. On the popular LVIS open-vocabulary detection benchmark, our approach sets a new state of the art of 37.6 mask APr using the common ViT-L backbone and public LAION dataset, and 40.5 mask APr using the DataComp-1B dataset, significantly outperforming the best existing approach by +3.7 mask APr at system level. On the COCO benchmark, we achieve very competitive 39.6 novel AP without pseudo labeling or weak supervision. In addition, we evaluate our approach on the transfer detection setup, where it demonstrates notable improvement over the baseline. Visualization reveals emerging object locality from the pretraining recipes compared to the baseline.

Figures (3)

• Figure 1: DITO overview (left): Existing image-text pretraining methods for open-vocabulary detection rovitkim2023contrastive only update the vision transformer backbone and finetune detector heads from scratch. We propose to pretrain both backbone and detector heads directly from the large-scale image-text paired data, without a need for pseudo labeling or box annotations. In open-vocabulary detection finetuning, we introduce a simple shifted-window learning method to produce more robust representations from the pretrained vision transformer. DITO prediction (right): LVIS results only show the novel categories (e.g. , bulldozer, fishbowl, subwoofer, heron). While Ego4D is a real-world and out-of-distribution data, many unseen objects are detected (e.g. , steel lid, sticker on the wall, recycle bin). Best viewed with zoom in.
• Figure 2: DITO method. Region-centric image-text pretraining (left): We train the detector heads (e.g. FPN li2022exploringlin2017feature, Faster RCNN head ren2015faster, and RPN ren2015faster) upon a ViT encoder backbone with multi-level image-text contrastive loss to bridge the gap between image-text pretraining and open-vocabulary detection. Shifted-window learning for detection (right): We roll the image and combines the shifted features with the original features to mitigate the bias of windowed attention grid li2022exploring, and produce more robust semantic representation.
• Figure 3: Visual-text similarity map. For each example, we show the paired image (left) and text (bottom) input, and the visual-text similarity map using the contrastive model baseline rovit backbone features (middle) or our region-centric pretraining features (right). We use Flickr30K (top row) and COCO Captions (bottom row) datasets.