Do Pre-trained Models Benefit Equally in Continual Learning?

TL;DR

This paper investigates whether pre-trained models add equal value across continual learning (CL) settings. It shows that benefits of pre-training vary substantially by CL algorithm, dataset complexity, and CL scenario, sometimes reversing algorithm rankings when initialization comes from pre-trained weights. A simple yet powerful baseline—Experience Replay (ER) with an ImageNet RN50 backbone plus a two-stage training pipeline—achieves strong, and often state-of-the-art, performance, underscoring the need to evaluate CL methods with pre-training in mind. Key takeaways include CLIP RN50's limited advantages in typical CL regimes and the pronounced forgetting reductions offered by self-supervised fine-tuning, suggesting hybrid approaches that blend self-supervision with supervised learning could be fruitful.

Abstract

Existing work on continual learning (CL) is primarily devoted to developing algorithms for models trained from scratch. Despite their encouraging performance on contrived benchmarks, these algorithms show dramatic performance drops in real-world scenarios. Therefore, this paper advocates the systematic introduction of pre-training to CL, which is a general recipe for transferring knowledge to downstream tasks but is substantially missing in the CL community. Our investigation reveals the multifaceted complexity of exploiting pre-trained models for CL, along three different axes, pre-trained models, CL algorithms, and CL scenarios. Perhaps most intriguingly, improvements in CL algorithms from pre-training are very inconsistent an underperforming algorithm could become competitive and even state-of-the-art when all algorithms start from a pre-trained model. This indicates that the current paradigm, where all CL methods are compared in from-scratch training, is not well reflective of the true CL objective and desired progress. In addition, we make several other important observations, including that CL algorithms that exert less regularization benefit more from a pre-trained model; and that a stronger pre-trained model such as CLIP does not guarantee a better improvement. Based on these findings, we introduce a simple yet effective baseline that employs minimum regularization and leverages the more beneficial pre-trained model, coupled with a two-stage training pipeline. We recommend including this strong baseline in the future development of CL algorithms, due to its demonstrated state-of-the-art performance.

Figures (5)

• Figure 1: (a) CL algorithms trained from scratch fail on Split CUB200, a more complex dataset than Split CIFAR100, which necessitates the use of pre-trained models (denoted as '+ RN18') that dramatically increase the accuracy of a wide spectrum of algorithms. (b) Different CL algorithms receive vastly different benefits from pre-trained models, and the superiority between algorithms changes. These findings suggest that it is critical for the community to develop CL algorithms with a pre-trained model and understand their behaviors. [Best viewed in color.]
• Figure 2: Forgetting of different models. Fine-tuning a pre-trained model in a self-supervised manner (with the SimCLR loss) largely decreases forgetting, compared with supervised fine-tuning. [Best viewed in color.]
• Figure 3: Comparison between CLIP R50 and ImageNet RN50. CLIP is better in the few-shot regime, while ImageNet RN50 prevails with plenty of data, as is the case for the CL scenarios considered in the paper. This explains the better performance when a CL learner is initialized from an ImageNet RN50 (cf. Table \ref{['tab:clip_vs_rn50']}). S- stands for Split. [Best viewed in color.]
• Figure 4: A simple second-staged offline training on memory data coupled with an ImageNet pre-trained ResNet50 turns a simple baseline into state of the art, suggesting the effectiveness of the proposed baseline. Note that SCR and iCaRL are the two best-performing methods when applied on the ImageNet pre-trained ResNet50. [Best viewed in color.]
• Figure A: Two-stage training CL pipeline. Most memory replay methods only perform learning during the streaming phase while on the contrary, GDumb only performs learning at the end of the stream (offline phase). Coupled with a pre-trained model, this simple two-stage pipeline that learns in both phases converts Experience Replay (ER) into a state-of-the-art approach (cf. Table \ref{['tab:two_stage']}).