Natural Mitigation of Catastrophic Interference: Continual Learning in Power-Law Learning Environments

TL;DR

The paper tackles catastrophic interference in continual learning by introducing naturalistic learning environments that mimic power-law task frequencies observed in humans, and evaluating how rehearsal-based training under these distributions mitigates forgetting. It defines power-law and exponential rehearsal schedules, tests them on key class-incremental benchmarks (SplitMNIST, SplitCIFAR-100, SplitTinyImageNet), and compares against standard baselines and popular CI mitigation methods, including regularization and rehearsal approaches. The results show that power-law rehearsal provides strong, often superior, mitigation of CI, especially as the number of phases grows, and can surpass many existing methods in harder tasks; combining power-law rehearsal with functional regularization like LwF can yield additional gains. The work argues for evaluating continual learning under naturalistic, power-law settings as a practical and model-agnostic baseline, suggesting a shift in how CI research should be conducted and validated for real-world learning scenarios.

Abstract

Neural networks often suffer from catastrophic interference (CI): performance on previously learned tasks drops off significantly when learning a new task. This contrasts strongly with humans, who can continually learn new tasks without appreciably forgetting previous tasks. Prior work has explored various techniques for mitigating CI and promoting continual learning such as regularization, rehearsal, generative replay, and context-specific components. This paper takes a different approach, one guided by cognitive science research showing that in naturalistic environments, the probability of encountering a task decreases as a power-law of the time since it was last performed. We argue that techniques for mitigating CI should be compared against the intrinsic mitigation in simulated naturalistic learning environments. Thus, we evaluate the extent of the natural mitigation of CI when training models in power-law environments, similar to those humans face. Our results show that natural rehearsal environments are better at mitigating CI than existing methods, calling for the need for better evaluation processes. The benefits of this environment include simplicity, rehearsal that is agnostic to both tasks and models, and the lack of a need for extra neural circuitry. In addition, we explore popular mitigation techniques in power-law environments to create new baselines for continual learning research.

Figures (2)

• Figure 1: Comparison of all the baselines and training environments on the SplitCIFAR-100 (above) and SplitTinyImageNet (below) class incremental learning scenarios with 10 phases. The values in the bracket indicate average test accuracy at the end of 10 phases. Note that EWC performs as poorly as the lower baseline across all phases for both datasets. LB: Lower Baseline, UB: Upper Baseline, EWC: Elastic Weight Consolidation, SI: Synaptic Intelligence, LwF: Learning without Forgetting, ER: Experience Replay, BIR: Brain-Inspired Replay, A-GEM: Averaged Gradient Episodic Memory, PL: Power-law, Exp: Exponential, iCaRL: Incremental Classifier and Representation Learning.
• Figure 2: Example of the Class Incremental Learning scenario (SplitMNIST dataset): In each phase, new classes are incrementally added to the training.