UADet: A Remarkably Simple Yet Effective Uncertainty-Aware Open-Set Object Detection Framework

TL;DR

Open-set object detection remains challenging due to unlabeled unknown objects that can be misclassified as background or partial unknowns. UADet introduces an uncertainty-aware framework that jointly models appearance uncertainty from the RPN and geometry uncertainty from IoU to generate soft labels for unlabeled negatives, improving unknown recall while preserving known-class accuracy. The method extends naturally to Open World Object Detection via exemplar replay-based fine-tuning and demonstrates strong gains across OSOD and OWOD benchmarks, including transformer-based backbones. The approach is simple to implement, scales with modern detectors, and highlights the value of uncertainty-guided supervision for robust open-set perception in real-world scenes.

Abstract

We tackle the challenging problem of Open-Set Object Detection (OSOD), which aims to detect both known and unknown objects in unlabelled images. The main difficulty arises from the absence of supervision for these unknown classes, making it challenging to distinguish them from the background. Existing OSOD detectors either fail to properly exploit or inadequately leverage the abundant unlabeled unknown objects in training data, restricting their performance. To address these limitations, we propose UADet, an Uncertainty-Aware Open-Set Object Detector that considers appearance and geometric uncertainty. By integrating these uncertainty measures, UADet effectively reduces the number of unannotated instances incorrectly utilized or omitted by previous methods. Extensive experiments on OSOD benchmarks demonstrate that UADet substantially outperforms previous state-of-the-art (SOTA) methods in detecting both known and unknown objects, achieving a 1.8x improvement in unknown recall while maintaining high performance on known classes. When extended to Open World Object Detection (OWOD), our method shows significant advantages over the current SOTA method, with average improvements of 13.8% and 6.9% in unknown recall on M-OWODB and S-OWODB benchmarks, respectively. Extensive results validate the effectiveness of our uncertainty-aware approach across different open-set scenarios.

Figures (6)

• Figure 1: A comparison of the proposed uncertainty-aware labeling strategy (bottom) with previous methods which typically assign hard labels (top) for unknown objects. The red boxes depict ground-truth boxes, while the yellow boxes show proposals that do not match the ground-truth boxes. The predicted scores for each negative proposal are also displayed. (a) Pure background: Negative proposals that correspond to the real background are misclassified as an unknown class. (b) Unknown object parts: Negative proposals representing partial parts of unknown objects are overconfidently classified as the unknown class. (c)(d) Neglected unknown objects: Negative proposals are misclassified as "background", resulting in the omission of unknown objects. In contrast, our proposed pseudo label strategy can assign more appropriate pseudo labels for the above cases.
• Figure 2: The comparison of unknown recall among RPN, OpenDet, and our proposed method. The evaluation was performed on three distinct data splits, i.e., VOC-COCO-{20,40,60} introduced in han2022expanding. Notably, our proposed method exhibited a significant improvement in terms of unknown recall when compared to the SOTA OSOD framework, OpenDet.
• Figure 3: The unknown score distribution histograms of the RPN in a closed-set trained Faster R-CNN model. The statistics is done on VOC-COCO-20 han2022expanding.
• Figure 4: An illustration of the necessity to introduce geometric uncertainty. We present four negative proposals generated by the RPN. App. and Geo. represent the appearance uncertainty and the geometry uncertainty. For simplicity, we set App.= $\hat{o}_{i,j}$, Geo.= $1 - u_{i,j}$, Both = $\text{App.} \times \text{Geo}$. Among these proposals, B, C, and D exhibit ambiguity. By incorporating geometric uncertainty, we mitigate the reliance solely on appearance uncertainty during the label assignment process, thereby mitigating the potential decline in performance for known classes.
• Figure 5: Qualitative comparisons between the OpenDet (top), top-k hard pseudo-labeling strategy (middle) and ours (bottom). All models are trained on VOC, and the detection results on COCO are visualized. It is important to note that for improved visualization, we apply NMS between known classes and the unknown class.