BSDP: Brain-inspired Streaming Dual-level Perturbations for Online Open World Object Detection

TL;DR

This paper takes the dual-level information of old samples as perturbations on new samples to make the model good at learning new knowledge without forgetting the old knowledge and proposes a simple plug-and-play method, called Brain-inspired Streaming Dual-level Perturbations (BSDP), to solve the OLOWOD problem.

Abstract

Humans can easily distinguish the known and unknown categories and can recognize the unknown object by learning it once instead of repeating it many times without forgetting the learned object. Hence, we aim to make deep learning models simulate the way people learn. We refer to such a learning manner as OnLine Open World Object Detection(OLOWOD). Existing OWOD approaches pay more attention to the identification of unknown categories, while the incremental learning part is also very important. Besides, some neuroscience research shows that specific noises allow the brain to form new connections and neural pathways which may improve learning speed and efficiency. In this paper, we take the dual-level information of old samples as perturbations on new samples to make the model good at learning new knowledge without forgetting the old knowledge. Therefore, we propose a simple plug-and-play method, called Brain-inspired Streaming Dual-level Perturbations(BSDP), to solve the OLOWOD problem. Specifically, (1) we first calculate the prototypes of previous categories and use the distance between samples and the prototypes as the sample selecting strategy to choose old samples for replay; (2) then take the prototypes as the streaming feature-level perturbations of new samples, so as to improve the plasticity of the model through revisiting the old knowledge; (3) and also use the distribution of the features of the old category samples to generate adversarial data in the form of streams as the data-level perturbations to enhance the robustness of the model to new categories. We empirically evaluate BSDP on PASCAL VOC and MS-COCO, and the excellent results demonstrate the promising performance of our proposed method and learning manner.

Figures (12)

• Figure 1: Overview of the OLOWOD problem. The model will see all samples in training sets only once. As the number of tasks increased, the model could recognize more different categories of objects without forgetting previously learned categories. The green boxes on the right indicate objects of known categories, while the red boxes mean that the objects belong to unknown categories.
• Figure 2: The figure illustrates the overall architecture module of our BSDP. The sign '$\oplus$' represents the addition operation according to certain weights and '$\otimes$' means similarity calculation. 'P' is short for prototype. Prototypes of all seen categories are calculated for sample selection strategies and our dual-level perturbations. In each task, samples are fed into RPN and RoI Pooling to obtain uniform sample features(process ①), and then into RoI head. After training, the model is used to extract the category features of bounding boxes to calculate prototypes, which is the process ②. Process ② is not executed during exemplars replay.
• Figure 3: Description of the calculation of prototypes for our BSDP and dual-level perturbations.
• Figure 4: Description of the feature-level perturbations of our BSDP. '$\otimes$' is similarity calculation and '$\oplus$' represents the addition operation according to certain weights. $W$ is the weight vector obtained after similarity calculation.
• Figure 5: Illustration of the data-level perturbations of our BSDP. $F$ represents the features of samples of each old category stored in addition after model training. We use the features of old categories $F$ to fit the given distributions, which is achieved by using the FitterCokelaer2014 library.