Incremental Object-Based Novelty Detection with Feedback Loop

TL;DR

This work addresses the challenge of object-based novelty detection in dynamic environments by introducing an incremental OND with a feedback loop. A lightweight OND module $M_{ND}$ sits on top of a pretrained detector $M_{OD}$ and is refined via human annotations in cloud sessions, while preserving $M_{OD}$ performance. The method leverages Supervised Contrastive Loss to structure latent space and a 1D BCE projection to produce id–ood scores, with incremental updates enabled by Experience Replay. Results on a new Test-Domain Incremental Benchmark show that iConP yields steady improvements in ood robustness as feedback accumulates, outperforming traditional baselines and demonstrating practical value for safety-critical deployment.

Abstract

Object-based Novelty Detection (ND) aims to identify unknown objects that do not belong to classes seen during training by an object detection model. The task is particularly crucial in real-world applications, as it allows to avoid potentially harmful behaviours, e.g. as in the case of object detection models adopted in a self-driving car or in an autonomous robot. Traditional approaches to ND focus on one time offline post processing of the pretrained object detection output, leaving no possibility to improve the model robustness after training and discarding the abundant amount of out-of-distribution data encountered during deployment. In this work, we propose a novel framework for object-based ND, assuming that human feedback can be requested on the predicted output and later incorporated to refine the ND model without negatively affecting the main object detection performance. This refinement operation is repeated whenever new feedback is available. To tackle this new formulation of the problem for object detection, we propose a lightweight ND module attached on top of a pre-trained object detection model, which is incrementally updated through a feedback loop. We also propose a new benchmark to evaluate methods on this new setting and test extensively our ND approach against baselines, showing increased robustness and a successful incorporation of the received feedback.

Figures (6)

• Figure 1: Our proposed setting for incremental object-based novelty detection with feedback loop. During deployment, a novelty detection function (ours is coined iConP) is responsible for detecting whether a feature of an instance, extracted by the object detection model represents an ood or an id object. These predictions, along with original pictures, are shared on the cloud with human annotators that provide weak labels in form of rejecting or accepting the model id/ood decisions. This feedback is then used by the model to increase its ood robustness. Comparison of predicted scores distribution with Max Logit (baseline - no feedback loop) and iConP - ours after few sessions of feedback are shown on the right.
• Figure 2: Our proposed approach. During training (left) we optimize a light-weight ND module to project features from a pretrained object detection model into a hyper sphere, where id instances are grouped together and far from ood samples; jointly a scoring function is optimized as a 1D projection of learned latent features with BCE. At test time (right) id samples would be assigned high score indicating an instance of a known object as opposed to ood samples.
• Figure 3: Novelty detection performance (FPR@95, lower is better) of Max Logit (dashed line), iBCE and iConP on the Test-Domain Incremental benchmark, evaluated after each session on $\mathit{G}^{holdout}$. iConP is capable of steady improvements as feedback is received contrary to iBCE.
• Figure 4: Our incremental training and evaluation protocol. At each training session $S_i$ a new group of id and ood samples $G_i$ is received. Evaluation is performed on a distinct group of id-ood samples, namely $G^{holdout}$.
• Figure 5: The id-ood scores distributions on $G^{holdout}$ from COCO-OpenImages benchmark predicted by (a) Max Logit, (b) iConP after the initial training session $S_0$, and (c) iConP after $S_4$. Starting with overlapping distribution with Max Logit, iConP separates the id-ood and largely benefits from the received feedback.