Boosting Open-Vocabulary Object Detection by Handling Background Samples

TL;DR

Open-vocabulary detectors using CLIP struggle with background samples, especially for oversized and partial regions. The authors propose BIRDet, featuring Background Information Modeling (BIM) to form a dynamic background embedding $t_{bg}(I)$ from top-$K$ scene prompts and Partial Object Suppression (POS) to remove partial-object detections via a threshold on Overlap Area Ratio, with a re-scoring mechanism that blends scene-derived background similarities into final scores. Empirical results on OV-COCO and OV-LVIS show consistent gains across multiple detectors and ablations validate BIM and POS as complementary, though gains on OV-LVIS are smaller due to higher category diversity and background bias. The work demonstrates a practical, plug-and-play improvement for CLIP-based OVOD by explicitly modeling and suppressing background influences, influencing future research on background-aware visual-language integration.

Abstract

Open-vocabulary object detection is the task of accurately detecting objects from a candidate vocabulary list that includes both base and novel categories. Currently, numerous open-vocabulary detectors have achieved success by leveraging the impressive zero-shot capabilities of CLIP. However, we observe that CLIP models struggle to effectively handle background images (i.e. images without corresponding labels) due to their language-image learning methodology. This limitation results in suboptimal performance for open-vocabulary detectors that rely on CLIP when processing background samples. In this paper, we propose Background Information Representation for open-vocabulary Detector (BIRDet), a novel approach to address the limitations of CLIP in handling background samples. Specifically, we design Background Information Modeling (BIM) to replace the single, fixed background embedding in mainstream open-vocabulary detectors with dynamic scene information, and prompt it into image-related background representations. This method effectively enhances the ability to classify oversized regions as background. Besides, we introduce Partial Object Suppression (POS), an algorithm that utilizes the ratio of overlap area to address the issue of misclassifying partial regions as foreground. Experiments on OV-COCO and OV-LVIS benchmarks demonstrate that our proposed model is capable of achieving performance enhancements across various open-vocabulary detectors.

Figures (5)

• Figure 1: (a) An example of oversized regions and (b) an example of partial regions. Green boxes are ground truths, while red dashed boxes are oversized regions and partial regions respectively. Oversized regions have excessive background information, while partial regions only contain part of objects.
• Figure 2: (a) Classification results on oversized regions of the CLIP RN50$\times$64 model radford2021learning. (b) Classification results on partial regions of the CLIP RN50$\times$64 model. The x-axis stands for the IoU range of regions, while the y-axis stands for classification probabilities. We measure the top-1 classification probability for ground truth categories and top-5 background classification probabilities respectively.
• Figure 3: Method overview. It consists of two novel modules: Background Information Modeling (BIM) and Partial Object Suppression (POS) algorithm. In addition to extracting foreground features using the CLIP image encoder, it classifies images into scene information and prompt it as background representations. After encoding, we use it as the background embedding for the RoI head and re-score classification scores to reduce bias caused by scene information. When initial detection results are available, we use the POS algorithm to suppress partial regions and obtain the final detection results.
• Figure 4: An example of the partial object suppression (POS) algorithm. After NMS, we sort candidate bounding boxes by scores and compute the overlap area ratio (OAR) between the boxes. We suppress boxes with OAR no less than $\Theta$ to obtain the refined detection results.
• Figure 5: Examples of qualitative results on OV-COCO. Blue detection results are base categories, while red detection results are novel categories. Our analysis demonstrates that BIRDet effectively reduces the misclassification of novel objects as either background or base categories and produces fewer redundant detections compared to the base detector (i.e. CLIPSelf wu2023clipself).