Towards Open-World Object-based Anomaly Detection via Self-Supervised Outlier Synthesis

TL;DR

This work tackles open-world object-based anomaly detection without relying on class labels by integrating an open-world object detector (OWOD) with unsupervised pseudo-classification and self-supervised outlier synthesis. The proposed OLN-SSOS framework learns pseudo-classes from object-level features, builds class-conditional Gaussians to sample virtual outliers, and uses an energy-based head to distinguish in-distribution objects from anomalies, with an optional OLN-FFS variant. Across diverse datasets and imaging modalities, including VOC/COCO, BDD/COCO, SIXRay10, LTDImaging, and DBF6, OLN-SSOS achieves state-of-the-art anomaly detection performance and demonstrates improved localization of unseen anomalies, aided by mask-based extensions. The approach offers robust, cross-modal open-world anomaly detection without class supervision and extends to instance segmentation, supporting practical applications in surveillance and security with strong generalisation capabilities.

Abstract

Object detection is a pivotal task in computer vision that has received significant attention in previous years. Nonetheless, the capability of a detector to localise objects out of the training distribution remains unexplored. Whilst recent approaches in object-level out-of-distribution (OoD) detection heavily rely on class labels, such approaches contradict truly open-world scenarios where the class distribution is often unknown. In this context, anomaly detection focuses on detecting unseen instances rather than classifying detections as OoD. This work aims to bridge this gap by leveraging an open-world object detector and an OoD detector via virtual outlier synthesis. This is achieved by using the detector backbone features to first learn object pseudo-classes via self-supervision. These pseudo-classes serve as the basis for class-conditional virtual outlier sampling of anomalous features that are classified by an OoD head. Our approach empowers our overall object detector architecture to learn anomaly-aware feature representations without relying on class labels, hence enabling truly open-world object anomaly detection. Empirical validation of our approach demonstrates its effectiveness across diverse datasets encompassing various imaging modalities (visible, infrared, and X-ray). Moreover, our method establishes state-of-the-art performance on object-level anomaly detection, achieving an average recall score improvement of over 5.4% for natural images and 23.5% for a security X-ray dataset compared to the current approaches. In addition, our method detects anomalies in datasets where current approaches fail. Code available at https://github.com/KostadinovShalon/oln-ssos.

Figures (21)

• Figure 1: Exemplar of in-distribution (green) and out-of-distribution (red) objects from across four diverse benchmark datasets with various imaging modalities.
• Figure 2: Our proposed architecture for open-world anomaly detection combines the object localisation network (OLN), unsupervised pseudo-classification (UPC) and anomaly detection via self-supervised outlier synthesis (SSOS).
• Figure 3: The qualitative results exemplify the effectiveness of our proposed approaches (OLN-SSOS/OLN-FFS) in detecting OoD/anomalous objects (in red bounding boxes).
• Figure 4: Qualitative results obtained from the DBF6 dataset utilising OLN-SSOS Mask.
• Figure 5: Ablation Studies