A Methodology-Oriented Study of Catastrophic Forgetting in Incremental Deep Neural Networks

TL;DR

This paper provides a methodology-oriented survey of catastrophic forgetting in incremental deep neural networks, organizing approaches into exemplar/memory-based and network-based categories. It examines three learning scenarios (task-, domain-, and class-based continual learning) and presents a structured mathematical overview of replay, regularization, and architectural methods used to mitigate CF. Key findings indicate replay-based methods (e.g., DGR, iCaRL) tend to be robust across scenarios, while domain- and class-based settings remain challenging for many regularization approaches, underscoring the value of task-aware architectures and rehearsal strategies. The work aims to guide researchers in comparing methods using standardized criteria and motivates future lifelong learning systems that can integrate external knowledge for continuous adaptation in real-world tasks.

Abstract

Human being and different species of animals having the skills to gather, transferring knowledge, processing, fine-tune and generating information throughout their lifetime. The ability of learning throughout their lifespan is referred as continuous learning which is using neurocognition mechanism. Consequently, in real world computational system of incremental learning autonomous agents also needs such continuous learning mechanism which provide retrieval of information and long-term memory consolidation. However, the main challenge in artificial intelligence is that the incremental learning of the autonomous agent when new data confronted. In such scenarios, the main concern is catastrophic forgetting(CF), i.e., while learning the sequentially, neural network underfits the old data when it confronted with new data. To tackle this CF problem many numerous studied have been proposed, however it is very difficult to compare their performance due to dissimilarity in their evaluation mechanism. Here we focus on the comparison of all algorithms which are having similar type of evaluation mechanism. Here we are comparing three types of incremental learning methods: (1) Exemplar based methods, (2) Memory based methods, and (3) Network based method. In this survey paper, methodology oriented study for catastrophic forgetting in incremental deep neural network is addressed. Furthermore, it contains the mathematical overview of impact-full methods which can be help researchers to deal with CF.

Figures (5)

• Figure 1: Multi-class classifier
• Figure 2: Catastrophic Forgetting Phenomenon
• Figure 3: Catastrophic Forgetting
• Figure 8: Average Validation Accuracy on split MNIST task dataset
• Figure 9: Average Validation Accuracy on p-MNIST task dataset