Detecting Unknown Objects via Energy-based Separation for Open World Object Detection

Abstract

In this work, we tackle the problem of Open World Object Detection (OWOD). This challenging scenario requires the detector to incrementally learn to classify known objects without forgetting while identifying unknown objects without supervision. Previous OWOD methods have enhanced the unknown discovery process and employed memory replay to mitigate catastrophic forgetting. However, since existing methods heavily rely on the detector's known class predictions for detecting unknown objects, they struggle to effectively learn and recognize unknown object representations. Moreover, while memory replay mitigates forgetting of old classes, it often sacrifices the knowledge of newly learned classes. To resolve these limitations, we propose DEUS (Detecting Unknowns via energy-based Separation), a novel framework that addresses the challenges of Open World Object Detection. DEUS consists of Equiangular Tight Frame (ETF)-Subspace Unknown Separation (EUS) and an Energy-based Known Distinction (EKD) loss. EUS leverages ETF-based geometric properties to create orthogonal subspaces, enabling cleaner separation between known and unknown object representations. Unlike prior energy-based approaches that consider only the known space, EUS utilizes energies from both spaces to better capture distinct patterns of unknown objects. Furthermore, EKD loss enforces the separation between previous and current classifiers, thus minimizing knowledge interference between previous and newly learned classes during memory replay. We thoroughly validate DEUS on OWOD benchmarks, demonstrating outstanding performance improvements in unknown detection while maintaining competitive known class performance.

Figures (4)

• Figure 1: Motivation and objectives of DEUS. (a) PCA visualization of proposal features: the baseline (top) entangles known, unknown, and background, while DEUS (bottom) achieves clear separation. (b) Objectives of DEUS: separating known from unknown proposals while distinguishing previous and current known classes.
• Figure 2: Overview of DEUS. ETF-Subspace Unknown Separation (EUS) utilizes a Simplex ETF to construct known and unknown spaces, where each space serves as an energy module to compute space energy scores, and Energy-based Known Distinction (EKD) loss is applied during the memory replay phase, where the classification branch is split into two sub-classifiers to calculate energy scores for previous and current tasks.
• Figure 3: (a) Per-class subspace score comparison between known (green) and unknown (red) for proposals matched with ground-truth. (b) Energy scores of proposals from each task computed by each task head (w/o vs. w/ EKD). Parentheses show changes with EKD.
• Figure 4: Qualitative comparison between OrthogonalDet and DEUS. Purple boxes indicate unknown predictions, while other colored boxes indicate known class predictions. The model was trained on Task 1, where cow is known and giraffe is unknown.