RECALL+: Adversarial Web-based Replay for Continual Learning in Semantic Segmentation

TL;DR

This work tackles catastrophic forgetting in continual semantic segmentation by introducing RECALL+, a web-based replay framework that operates without storing past data. The approach combines adversarial image selection and semantic-content thresholding to curate web-replay samples, and augments these with background and knowledge inpainting to reduce background shift and propagate current-task information. Empirical results on Pascal VOC 2012 and ADE20K show improved retention of old knowledge and strong performance across long sequences of incremental tasks, outperforming many existing methods in multi-step scenarios. The work demonstrates that unsupervised web data can serve as a practical, scalable source for replay without extra labeling, with a scalable discriminator-based filtering and per-class content thresholds enabling robust continual learning.

Abstract

Catastrophic forgetting of previous knowledge is a critical issue in continual learning typically handled through various regularization strategies. However, existing methods struggle especially when several incremental steps are performed. In this paper, we extend our previous approach (RECALL) and tackle forgetting by exploiting unsupervised web-crawled data to retrieve examples of old classes from online databases. In contrast to the original methodology, which did not incorporate an assessment of web-based data, the present work proposes two advanced techniques: an adversarial approach and an adaptive threshold strategy. These methods are utilized to meticulously choose samples from web data that exhibit strong statistical congruence with the no longer available training data. Furthermore, we improved the pseudo-labeling scheme to achieve a more accurate labeling of web data that also considers classes being learned in the current step. Experimental results show that this enhanced approach achieves remarkable results, particularly when the incremental scenario spans multiple steps.

Figures (9)

• Figure 1: Replay images of previously seen classes are retrieved by a web crawler and then filtered by a domain discriminator, after which the network is incrementally trained with a mixture of new and replay data.
• Figure 2: Comparison of predictions between the model trained with knowledge self-inpainting and without inpainting. Inpainting the replay images accelerates convergence and improves segmentation accuracy.
• Figure 3: Overview of RECALL+: Class labels from the past steps are retrieved by Source Block, which consists of a domain-discriminator and filters the duplicate and near-duplicate images for each class. Then these selected images are further filtered by a CDF-based thresholding strategy. Finally, the segmentation network is incrementally trained with both replay data and new class data.
• Figure 4: Background self-inpainting process and knowledge painting process. At step $\textit{k}$, the background self-inpainting technique updates the past knowledge (i.e. ,horse) on the current step training images before the training step starts. During training, the knowledge self-inpainting updates the label information of the classes being learned (i.e., person) to the web downloaded images.
• Figure 5: Examples of filtered and accepted images according to our adversarial image selection strategy.