Bridge Past and Future: Overcoming Information Asymmetry in Incremental Object Detection

TL;DR

This work tackles information asymmetry in incremental object detection (IOD), where images can contain objects from past, present, and future categories, causing forgetting and inconsistent learning. It introduces Bridge Past and Future (BPF) to align supervision across stages by leveraging pseudo labels from the past and by excluding probable future objects from negative samples, and Distillation with Future (DwF) which uses two teachers (the old model and an intermediate current-model expert) to distill knowledge across all categories in a class-aware manner. The approach is evaluated in memory-free settings on PASCAL VOC and MS COCO, where BPF + DwF outperforms state-of-the-art methods across multiple single- and multi-step incremental scenarios, with significant gains on old-class detection and competitive gains on new-class learning. The results demonstrate the practical impact of incorporating future and past information into the training loop, enabling more robust continual adaptation for object detection without storing historical data. All mathematical expressions related to the method are presented within $...$ delimiters to preserve precision.

Abstract

In incremental object detection, knowledge distillation has been proven to be an effective way to alleviate catastrophic forgetting. However, previous works focused on preserving the knowledge of old models, ignoring that images could simultaneously contain categories from past, present, and future stages. The co-occurrence of objects makes the optimization objectives inconsistent across different stages since the definition for foreground objects differs across various stages, which limits the model's performance greatly. To overcome this problem, we propose a method called ``Bridge Past and Future'' (BPF), which aligns models across stages, ensuring consistent optimization directions. In addition, we propose a novel Distillation with Future (DwF) loss, fully leveraging the background probability to mitigate the forgetting of old classes while ensuring a high level of adaptability in learning new classes. Extensive experiments are conducted on both Pascal VOC and MS COCO benchmarks. Without memory, BPF outperforms current state-of-the-art methods under various settings. The code is available at https://github.com/iSEE-Laboratory/BPF.

Figures (6)

• Figure 1: Challenges faced by the IOD task. Classes from previous, current, and potential future stages may appear arbitrarily in the current training stage, while only the annotations of current classes are available to train the detector. Best viewed in color.
• Figure 2: The overall framework of our method. The top side illustrates the Bridge Past and Future (BPF) procedure, which identifies objects of past classes and excludes several potential objects of future classes to ensure consistent optimization during the entire training process. The bottom side shows the Distillation with Future (DwF) process, which employs both the old model $\mathcal{M}_{t-1}$ adept at detecting old categories and the interim model $\mathcal{M}_{t}^{im}$ trained on $\mathcal{D}_t$ and specialized in new categories, to conduct a comprehensive distillation across all categories for the current model $\mathcal{M}_t$.
• Figure 3: Overview of Bridge Past and Future. We adopt the previous model $\mathcal{M}_{t-1}$ to predict some pseudo labels for past classes to complement their missing supervision in the current stage. Additionally, we exclude several proposals that are likely to be an object but are not included in the current ground truth and pseudo labels from the background to avoid classifying them into background mistakenly.
• Figure 4: The illustration of Distillation with Future strategy. An intermediate teacher model trained on the current dataset is used to compensate for the lack of current class information in the old model. For proposals overlapping ground truth of the current stage, since the intermediate teacher is specialized in detecting current classes, we directly inherit its probabilities on current classes and use the old model to enrich its background probability with old class knowledge. On the contrary, for proposals that do not overlap with GT, the old model is preferred, and the intermediate model is used as compensation. Combining two teachers makes the distillation class by class.
• Figure 5: Qualitative results for the model trained under the 10+10 setting on the VOC 2007 test set. 'boat', 'cat', 'chair', and 'car' are old classes from the first stage, and 'person' and 'dog' are the classes from the second stage. Compared with MMA (bottom row), our BPF (top row) can produce reliable predictions on both old and new classes.