Enhancing Novel Object Detection via Cooperative Foundational Models

TL;DR

The paper tackles novel object detection by turning closed-set detectors into open-set detectors through a cooperative framework that combines pre-trained foundational models (CLIP, SAM) with open-set detectors (e.g., GDINO). The method initializes proposals from existing detectors, labels unknowns with CLIP using Synonym Averaged Embedding Generators, and refines predictions with SAM and a Score Refinement Module, achieving strong zero-shot performance. Empirical results on LVIS and COCO OVD show state-of-the-art Novel AP and favorable Known AP, with substantial gains over RNCDL and competitive Open-Vocabulary Detection benchmarks. The approach is modular and training-free, offering practical benefits for long-tail object distributions, though speed and data leakage remain areas for future work, and extending to instance segmentation is a natural next step.

Abstract

In this work, we address the challenging and emergent problem of novel object detection (NOD), focusing on the accurate detection of both known and novel object categories during inference. Traditional object detection algorithms are inherently closed-set, limiting their capability to handle NOD. We present a novel approach to transform existing closed-set detectors into open-set detectors. This transformation is achieved by leveraging the complementary strengths of pre-trained foundational models, specifically CLIP and SAM, through our cooperative mechanism. Furthermore, by integrating this mechanism with state-of-the-art open-set detectors such as GDINO, we establish new benchmarks in object detection performance. Our method achieves 17.42 mAP in novel object detection and 42.08 mAP for known objects on the challenging LVIS dataset. Adapting our approach to the COCO OVD split, we surpass the current state-of-the-art by a margin of 7.2 $ \text{AP}_{50} $ for novel classes. Our code is available at https://rohit901.github.io/coop-foundation-models/ .

Figures (6)

• Figure 1: Comparison of top-10 predictions: (a) RNCDL fomenko2022learning can result in imprecise localization and misclassification (e.g., basket, apple), versus (b) our open-set Mask-RCNN, demonstrating accurate detection and categorization of unique objects in the scene.
• Figure 2: Comparative analysis of object detection methods on lvis_v1_val_subset dataset. The closed-set Mask-RCNN does not detect novel classes, however, the performance consistently improves when combined with our cooperative mechanism integrating different foundational models.
• Figure 3: Our proposed cooperative mechanism integrates pre-trained foundational models such as CLIP, SAM, and GDINO with a Mask-RCNN model in order to identify and semantically label both known and novel objects. These foundational model interacts using different components including Initialization, Unknown Object Labelling, and Refinement to refine and categorize objects.
• Figure 4: Object detection "mAP" performance on the lvis_val_subset dataset. "Grounding Dino" and "Mask-RCNN-V1" models were evaluated after considering the first 300, second 300, and the first 600 boxes.
• Figure 5: Ablation of our refinement component on the lvis_val_subset dataset. The figure reports the Average Precision (AP) for both Novel and Known objects.