Architecture Matters in Continual Learning

TL;DR

This paper reveals that the neural architecture chosen for continual learning can dramatically affect both learning and retention, challenging the common practice of fixing architecture while improving algorithms. Through large-scale experiments across CIFAR-100, ImageNet-1K, and Rotated MNIST, the authors show that simple CNNs can rival or surpass more complex models by achieving lower forgetting without sacrificing learning, and that width, pooling, and normalization choices often matter more than sheer parameter count. They dissect the impact of width/depth, BN, skip connections, pooling, GAP, and attention heads, offering concrete guidelines (e.g., avoid GAP, favor max pooling, add BN and Skip connections) to improve retention with modest computational overhead. The work argues for a paradigm that treats architecture as a central design variable in CL, inviting future theoretical and empirical studies to engineer architectures with inductive biases tailored to distribution shifts and task sequences, alongside algorithmic advances.

Abstract

A large body of research in continual learning is devoted to overcoming the catastrophic forgetting of neural networks by designing new algorithms that are robust to the distribution shifts. However, the majority of these works are strictly focused on the "algorithmic" part of continual learning for a "fixed neural network architecture", and the implications of using different architectures are mostly neglected. Even the few existing continual learning methods that modify the model assume a fixed architecture and aim to develop an algorithm that efficiently uses the model throughout the learning experience. However, in this work, we show that the choice of architecture can significantly impact the continual learning performance, and different architectures lead to different trade-offs between the ability to remember previous tasks and learning new ones. Moreover, we study the impact of various architectural decisions, and our findings entail best practices and recommendations that can improve the continual learning performance.

Figures (4)

• Figure 1: Split CIFAR-100: (a) While compared to naive fine-tuning, continual learning algorithms such as EWC and ER improve the performance, a simple modification to the architecture (removing global average pooling (GAP) layer) can match the performance of ER with a replay size of 1000 examples. (b) and (c) Different architectures lead to very different continual learning performance levels in terms of accuracy and forgetting. This work will investigate the reasons behind these gaps and provide insights into improving architectures.
• Figure 2: Evolution of average accuracy for various architectures on (a) Split CIFAR-100: CNNs have smaller forgetting than other architectures while WideResNets have the highest learning accuracy, and (b) Split ImageNet-1K WideResNets and ResNets have higher learning accuracy than CNNs and ViTs. However, the latter has smaller forgetting.
• Figure 3: BN statistics for the first layer of CNN$\times 4$ on Split CIFAR-100: the statistics do not change significantly throughout the continual learning experience.
• Figure 4: BN statistics for the first layer of MLP-128 on Permuted MNIST: the statistics change more compared to Fig. \ref{['fig:cifar-bn1']}