Learning Background Prompts to Discover Implicit Knowledge for Open Vocabulary Object Detection

TL;DR

Open vocabulary object detection (OVD) suffers from poor background interpretation and overfitting, hampering recognition of novel categories. The authors propose Learning Background Prompts (LBP), a modular framework comprising Background Category-specific Prompt (BCP), Background Object Discovery (BOD), and Inference Probability Rectification (IPR) to harvest implicit background knowledge without requiring priors about novel classes. BCP discovers background-underlying categories with learnable prompts, BOD online discovers objects via CLIP-based pseudo-labeling, and IPR mitigates overlaps between background concepts and novel categories during inference, all while preserving CLIP encoders. On OV-COCO and OV-LVIS, LBP achieves state-of-the-art results, improves novel-category detection, and demonstrates robustness to background bias, underscoring the value of leveraging implicit background knowledge for practical OVD systems.

Abstract

Open vocabulary object detection (OVD) aims at seeking an optimal object detector capable of recognizing objects from both base and novel categories. Recent advances leverage knowledge distillation to transfer insightful knowledge from pre-trained large-scale vision-language models to the task of object detection, significantly generalizing the powerful capabilities of the detector to identify more unknown object categories. However, these methods face significant challenges in background interpretation and model overfitting and thus often result in the loss of crucial background knowledge, giving rise to sub-optimal inference performance of the detector. To mitigate these issues, we present a novel OVD framework termed LBP to propose learning background prompts to harness explored implicit background knowledge, thus enhancing the detection performance w.r.t. base and novel categories. Specifically, we devise three modules: Background Category-specific Prompt, Background Object Discovery, and Inference Probability Rectification, to empower the detector to discover, represent, and leverage implicit object knowledge explored from background proposals. Evaluation on two benchmark datasets, OV-COCO and OV-LVIS, demonstrates the superiority of our proposed method over existing state-of-the-art approaches in handling the OVD tasks.

Figures (5)

• Figure 1: An example to illustrate previous and our designs in background interpretation. Conventional designs use a single background embedding to push the RoI embedding away from the CLIP embedding. DetPro du2022learning proposes to uniformly push the RoI embedding away when the CLIP embedding nears a base class embedding, leading to a loss of class relation. Our LBP, on the other hand, learns multiple background underlying class embeddings, effectively preserving class relations and alleviating loss conflict. "Distillation loss" uses knowledge distillation to align visual features encoded by the decoder with CLIP embeddings, while "Background classification loss" refers to the classification loss for background proposals.
• Figure 2: An overview of the proposed LBP approach, consisting of three modules: Background Category-specific Prompt (BCP), Background Object Discovery (BOD), and Inference Probability Rectification (IPR). During training, BCP is first proposed to discover and represent background underlying categories, estimated from background proposals, with learnable background category-specific contexts. Then, BOD is presented to employ $k$-means clustering on background proposals across all images to harness implicit objects explored from background underlying knowledge. During inference, IPR is introduced to rectify probability scores of novel categories provided, by loosening their conceptual overlaps with background underlying categories estimated from background proposals.
• Figure 3: Visualizations of feature distributions for novel category proposals. Note that proposals are selected for those that exhibit significant IoU overlap with the ground truth boxes of novel categories. Different colors denote distinct categories. (a) and (b) showcase the feature distributions generated by BARON wu2023aligning and our LBP, respectively. Compared to BARON, our LBP algorithm leads to more compact distributions for the same novel category representations.
• Figure 4: Visualizations of representative proposals for three background underlying categories estimated during training. They illustrate the potential conceptual overlaps between those background underlying categories and novel categories during inference on OV-COCO.
• Figure 5: Visualization of the distributions of contextual embeddings of base categories ${\mathcal{C}_b}$, novel categories ${\mathcal{C}_u}$, and background underlying categories ${\mathcal{C}_O}$ in OV-COCO task. Here, we harness the magnitude of the shrinking factor, i.e., $( 1 - \sum_{c^{\prime\prime}\in {\mathcal{C}}_u}P(c^{\prime\prime}|x, c^{\prime}))$ in Eq. (18), to showcase the semantic similarity or conceptual overlap between estimated background categories and inference novel categories. The color bar represents the relationships between $( 1 - \sum_{c^{\prime\prime}\in {\mathcal{C}}_u}P(c^{\prime\prime}|x, c^{\prime}))$ for each $c^{\prime} \in \mathcal{C}_O$ and the shades of blue, where darker shades indicate lower degrees of conceptual overlaps and vice versa.