Rethinking Open-Set Object Detection: Issues, a New Formulation, and Taxonomy

TL;DR

This paper critiques the prevailing OSOD formulations and introduces OSOD-III, a practical open-set detection framework that is open at the class level but closed at the super-class level. By benchmarking OSOD-III on Open Images, CUB200, and MTSD, the authors show that existing OSOD-II methods poorly detect unknowns when evaluated with standard AP metrics, and that simple baselines can be competitive. They also reveal evaluation pitfalls in prior metrics (A-OSE, WI) and provide a taxonomy clarifying OSOD terminology. The work highlights a clear path for progress: define targets within a superclass, evaluate with AP on unknowns, and investigate detector behavior and NMS strategies to reduce confusion between known and unknown detections. Overall, OSOD-III offers a realistic, testable formulation that is likely to steer practical progress in open-set object detection.

Abstract

Open-set object detection (OSOD), a task involving the detection of unknown objects while accurately detecting known objects, has recently gained attention. However, we identify a fundamental issue with the problem formulation employed in current OSOD studies. Inherent to object detection is knowing "what to detect," which contradicts the idea of identifying "unknown" objects. This sets OSOD apart from open-set recognition (OSR). This contradiction complicates a proper evaluation of methods' performance, a fact that previous studies have overlooked. Next, we propose a novel formulation wherein detectors are required to detect both known and unknown classes within specified super-classes of object classes. This new formulation is free from the aforementioned issues and has practical applications. Finally, we design benchmark tests utilizing existing datasets and report the experimental evaluation of existing OSOD methods. The results show that existing methods fail to accurately detect unknown objects due to misclassification of known and unknown classes rather than incorrect bounding box prediction. As a byproduct, we introduce a taxonomy of OSOD, resolving confusion prevalent in the literature. We anticipate that our study will encourage the research community to reconsider OSOD and facilitate progress in the right direction.

Figures (9)

• Figure 1: Illustration of OSOD-I, -II, and -III with image examples (top) and object class space (bottom). OSOD-I: Detect the known objects without being distracted by unknown objects. OSOD-II: Detect known and unknown objects as such, although 'objectness'---what should or should not be a detection target---is ambiguous unless explicitly defined. OSOD-III: Detect known and unknown objects belonging to the same super-class as such.
• Figure 2: Example images showing that "object" is an ambiguous concept. It is impractical to cover an unlimited range of object instances with a finite set of predefined categories.
• Figure 3: A-OSE (a) and WI (b) of different methods at different detector operating points. Smaller values mean better performance for both metrics. The horizontal axis indicates the confidence threshold for selecting bounding box candidates. Methods' ranking varies on the choice of the threshold.
• Figure 4: Example outputs of OpenDet Opendet and our baseline method with Faster RCNN FasterRCNN for Open Images with Animal and Vehicle super-classes and MTSD, respectively. Red and blue boxes indicate detected unknown-class and known-class objects, respectively; "Unk" means "unknown".
• Figure 5: Detection accuracy at various IoU thresholds for NMS between known and unknown predictions: mAP for known classes and AP for unknown. The results for (a) CUB200 and (b) MTSD.