Demystifying Catastrophic Forgetting in Two-Stage Incremental Object Detector

TL;DR

This work reveals that catastrophic forgetting in two-stage incremental object detectors predominantly arises from the RoI Head classifier, while the RPN and regressors show robustness. It introduces NSGP-RePRE, a targeted framework combining Regional Prototype Replay (coarse semantic centers and fine-grained intra-class prototypes) with Null Space Gradient Projection to prevent feature-drifts in the backbone. The approach achieves state-of-the-art results on VOC and COCO across various single- and multi-step incremental settings, and provides principled guidance for mitigating forgetting in incremental object detection. The findings offer concrete insights into component-specific forgetting and practical strategies to balance stability and plasticity in IOD systems.

Abstract

Catastrophic forgetting is a critical chanllenge for incremental object detection (IOD). Most existing methods treat the detector monolithically, relying on instance replay or knowledge distillation without analyzing component-specific forgetting. Through dissection of Faster R-CNN, we reveal a key insight: Catastrophic forgetting is predominantly localized to the RoI Head classifier, while regressors retain robustness across incremental stages. This finding challenges conventional assumptions, motivating us to develop a framework termed NSGP-RePRE. Regional Prototype Replay (RePRE) mitigates classifier forgetting via replay of two types of prototypes: coarse prototypes represent class-wise semantic centers of RoI features, while fine-grained prototypes model intra-class variations. Null Space Gradient Projection (NSGP) is further introduced to eliminate prototype-feature misalignment by updating the feature extractor in directions orthogonal to subspace of old inputs via gradient projection, aligning RePRE with incremental learning dynamics. Our simple yet effective design allows NSGP-RePRE to achieve state-of-the-art performance on the Pascal VOC and MS COCO datasets under various settings. Our work not only advances IOD methodology but also provide pivotal insights for catastrophic forgetting mitigation in IOD. Code is available at \href{https://github.com/fanrena/NSGP-RePRE}{https://github.com/fanrena/NSGP-RePRE} .

Figures (18)

• Figure 4: The overall architecture of our NSGP-RePRE framework. This framework incorporates RePRE to mitigate forgetting within the RoI Head's classifier. NSGP is introduced to counteract the shifts induced by the evolving feature extractor.
• Figure 5: mAP of different model on $\mathcal{D}_1^{test}$ in VOC(5-5) settings. To better demonstrate the impact of our method on the classifier, $P_1$ is fixed to all models. Fixed cls indicates the models classification results is designated by ${\cal M}_1$.
• Figure 6: Plot: Results of ${\cal M}_{joint}$ after removing high-quality proposals with varying IoU threshold. Bar: The distribution of the proposals generated with ${\cal M}_{joint}$ over IoU. The number on the bar indicates the count of proposals.
• Figure 7: An overview of NSGP.
• Figure : (a) RPN's recall on $\mathcal{D}_1^{test}$.