PANDAS: Prototype-based Novel Class Discovery and Detection

TL;DR

PANDAS introduces a prototype-based framework for novel class discovery and detection that extends a base detector with unlabeled novel-class prototypes learned via clustering. It uses base prototypes computed from labeled data and novel prototypes learned in a discovery phase, enabling a distance-based classifier to detect both base and novel classes without storing base images or knowing the number of novel classes. The approach achieves competitive or superior results to state-of-the-art RNCDL on VOC and COCO→LVIS benchmarks, with substantially lower compute times and memory requirements. The method demonstrates practical applicability by delivering real-time inference and offering avenues for semantic labeling via Hungarian matching or vision-language models, contributing a scalable, efficient paradigm for open-world detection with minimal manual intervention.

Abstract

Object detectors are typically trained once and for all on a fixed set of classes. However, this closed-world assumption is unrealistic in practice, as new classes will inevitably emerge after the detector is deployed in the wild. In this work, we look at ways to extend a detector trained for a set of base classes so it can i) spot the presence of novel classes, and ii) automatically enrich its repertoire to be able to detect those newly discovered classes together with the base ones. We propose PANDAS, a method for novel class discovery and detection. It discovers clusters representing novel classes from unlabeled data, and represents old and new classes with prototypes. During inference, a distance-based classifier uses these prototypes to assign a label to each detected object instance. The simplicity of our method makes it widely applicable. We experimentally demonstrate the effectiveness of PANDAS on the VOC 2012 and COCO-to-LVIS benchmarks. It performs favorably against the state of the art for this task while being computationally more affordable.

Figures (14)

• Figure 1: This paper tackles the task of novel class discovery and detection. We start from an initial detector, trained for base classes, and aim to extend it by automatically discovering novel classes from a set of unannotated images (i.e., the Discovery set). Novel classes are added to the detector, on the fly, without any manual intervention. The updated detector must then detect both base classes and novel classes in new test images. Note that this example only shows novel discovered classes for clarity.
• Figure 2: Overview of our PANDAS method. Given an annotated training set and a detector trained for Base classes on that training set, we compute Base prototypes. Then components from this detector are used during the discovery phase to compute prototypes for the Novel classes using clustering (top). During inference, similarity scores of new box features to these prototypes are computed to provide predicted scores (bottom).
• Figure 3: Qualitative results on VOC. Detections output by PANDAS. Base classes include "bottle" and "cat". Novel classes include "dog", "motorbike", "pottedplant", and "sheep". Additional examples on VOC are in Sec. \ref{['supp:timing']}.
• Figure 4: NCD Performance vs. Clusters on VOC. We study NCD performance as a function of the number of novel clusters. Each curve is averaged over three runs with shaded standard deviations.
• Figure 5: Clusters on COCO$\rightarrow$LVIS. For three discovered clusters, we show the five boxes closest to cluster centers and indicate the class name they were mapped to during assignment.