Group and Exclusive Sparse Regularization-based Continual Learning of CNNs

TL;DR

Catastrophic forgetting in continual learning with fixed-capacity CNNs is addressed by GESCL, which couples a stability regularizer on important past filters with a plasticity regularizer based on group and exclusive sparsity to sparsify unimportant filters for future tasks. The optimization uses proximal gradient descent, with adaptive filter importance guiding pruning and reinitialization to maintain capacity. Filter importance is updated across tasks via a cumulative score, and unneeded filters are reinitialized while their outgoing connections are zeroed to prevent interference. Experiments on SVHN, CIFAR-10/100, and ImageNet-50 demonstrate that GESCL achieves strong forgetting resistance and competitive accuracy relative to state-of-the-art baselines, while using fewer parameters and without storing past data.

Abstract

We present a regularization-based approach for continual learning (CL) of fixed capacity convolutional neural networks (CNN) that does not suffer from the problem of catastrophic forgetting when learning multiple tasks sequentially. This method referred to as Group and Exclusive Sparsity based Continual Learning (GESCL) avoids forgetting of previous tasks by ensuring the stability of the CNN via a stability regularization term, which prevents filters detected as important for past tasks to deviate too much when learning a new task. On top of that, GESCL makes the network plastic via a plasticity regularization term that leverage the over-parameterization of CNNs to efficiently sparsify the network and tunes unimportant filters making them relevant for future tasks. Doing so, GESCL deals with significantly less parameters and computation compared to CL approaches that either dynamically expand the network or memorize past tasks' data. Experiments on popular CL vision benchmarks show that GESCL leads to significant improvements over state-of-the-art method in terms of overall CL performance, as measured by classification accuracy as well as in terms of avoiding catastrophic forgetting.

Figures (4)

• Figure 1: Processing unimportant filters. Filter $2$ in layer $i$ ($\mathcal{F}_{i,2}$) is unimportant. Consequently, its kernel weights get reinitialized and channel $2$ of all filters in layer $i+1$ ($\mathcal{F}_{i+1,:2}$) is zeroed. The same applies to $\mathcal{F}_{i,4}$, which is also unimportant.
• Figure 2: Evaluation of catastrophic forgetting by measuring performance retention on the Initial task. Results show for each dataset how the classification accuracy of the first task evolves as further tasks are being learnt. Overall, GESCL show the strongest resilience against catastrophic forgetting. When facing new tasks, the performance it achieves on the initial task does not degrade as in concurrent approaches.
• Figure 3: Evaluation of catastrophic forgetting by measuring performance retention on all task for the CIFAR-10/100 dataset. Results show how the test accuracy of each task evolves as further tasks are being learnt. Overall, GESCL suffers less from catastrophic forgetting.
• Figure 4: Evaluation of average test accuracy during the continual learning experiment of three datasets.