Enhancing Open-Vocabulary Object Detection through Multi-Level Fine-Grained Visual-Language Alignment

TL;DR

This work addresses open-vocabulary object detection by bridging CLIP-based visual-language priors with detection architectures. It introduces VLDet, featuring a Visual-Language Encoder, a Visual-Language Pyramid Upscale Block (VL-PUB) for multi-scale features, and a Sigmoid-based Visual-Language RPN (SigRPN) to propose objects across categories. The model employs three alignment losses—$L_{ICL}$ (image-caption), $L_{AAL}$ (anchor-text binary), and $L_{RAL}$ (region-text)—to achieve fine-grained, multi-level visual-language alignment, enabling strong generalization to novel categories. Evaluated on COCO2017 and LVIS, VLDet achieves state-of-the-art Novel-AP (58.7 on COCO2017 and 24.8 on LVIS) and exhibits robust zero-shot and fine-tuned performance, illustrating the practical impact of integrating pre-aligned CLIP encoders with open-world detection pipelines.

Abstract

Traditional object detection systems are typically constrained to predefined categories, limiting their applicability in dynamic environments. In contrast, open-vocabulary object detection (OVD) enables the identification of objects from novel classes not present in the training set. Recent advances in visual-language modeling have led to significant progress of OVD. However, prior works face challenges in either adapting the single-scale image backbone from CLIP to the detection framework or ensuring robust visual-language alignment. We propose Visual-Language Detection (VLDet), a novel framework that revamps feature pyramid for fine-grained visual-language alignment, leading to improved OVD performance. With the VL-PUB module, VLDet effectively exploits the visual-language knowledge from CLIP and adapts the backbone for object detection through feature pyramid. In addition, we introduce the SigRPN block, which incorporates a sigmoid-based anchor-text contrastive alignment loss to improve detection of novel categories. Through extensive experiments, our approach achieves 58.7 AP for novel classes on COCO2017 and 24.8 AP on LVIS, surpassing all state-of-the-art methods and achieving significant improvements of 27.6% and 6.9%, respectively. Furthermore, VLDet also demonstrates superior zero-shot performance on closed-set object detection.

Figures (5)

• Figure 1: With multi-level fine-grained visual-language alignment, VLDet accurately detects objects from LVIS categories to user-specified ones.
• Figure 2: With multi-level visual-language alignment, VLDet is a unified network which can effectively exploit enriched visual-language semantic space from pre-aligned single-scale backbones while also providing pyramid image features for improved performance in OVD.
• Figure 3: (a) $\mathcal{L}_{ICL}$ for image-wise visual-language alignment is computed before VL-PUB module. After the fusion of image feature and text feature, VL-PUB generates deeper text feature and pyramid image feature at various scales. (b) SigRPN computes $\mathcal{L}_{AAL}$ for fine-grained visual-language alignment on the general difference of Background and Object of any class.
• Figure 4: Comparison between YOLO-World and VLDet, VLDet detects objects (e.g. "black turtle") more accurately.
• Figure 5: Image-level Contrastive Loss, $\mathcal{L}_{ICL}$ performs the best with mini-batch size of 8.