SR-CIS: Self-Reflective Incremental System with Decoupled Memory and Reasoning

TL;DR

SR-CIS addresses catastrophic forgetting in class-incremental learning by decoupling memory from reasoning through a deconstructed architecture of Complementary Memory Module (CMM) and Complementary Inference Module (CIM). It leverages task-specific LoRA-based parameter memory and prototype-based representation memory, a Scenario Replay Module with textual scenario descriptions, and a Confidence-Aware Online Anomaly Detection (CA-OAD) mechanism to switch between fast and slow inference. Memory is reshaped periodically by replaying scenarios via SDXL and merging memories into a Long-Term Memory (LTM), balancing plasticity and memory stability under restricted storage and data resources. Empirically, SR-CIS achieves state-of-the-art or competitive performance on CIFAR100, DomainNet, and ImageNet-R, with substantial gains when memory restructuring is applied, and demonstrates robustness in few-shot settings. This work offers a practical, memory-centric blueprint for continual learning inspired by human memory systems and CLS theory.

Abstract

The ability of humans to rapidly learn new knowledge while retaining old memories poses a significant challenge for current deep learning models. To handle this challenge, we draw inspiration from human memory and learning mechanisms and propose the Self-Reflective Complementary Incremental System (SR-CIS). Comprising the deconstructed Complementary Inference Module (CIM) and Complementary Memory Module (CMM), SR-CIS features a small model for fast inference and a large model for slow deliberation in CIM, enabled by the Confidence-Aware Online Anomaly Detection (CA-OAD) mechanism for efficient collaboration. CMM consists of task-specific Short-Term Memory (STM) region and a universal Long-Term Memory (LTM) region. By setting task-specific Low-Rank Adaptive (LoRA) and corresponding prototype weights and biases, it instantiates external storage for parameter and representation memory, thus deconstructing the memory module from the inference module. By storing textual descriptions of images during training and combining them with the Scenario Replay Module (SRM) post-training for memory combination, along with periodic short-to-long-term memory restructuring, SR-CIS achieves stable incremental memory with limited storage requirements. Balancing model plasticity and memory stability under constraints of limited storage and low data resources, SR-CIS surpasses existing competitive baselines on multiple standard and few-shot incremental learning benchmarks.

Figures (5)

• Figure 1: The schematic diagram of an ideal complementary learning system.
• Figure 2: The schematic diagram of an ideal complementary learning system.
• Figure 3: The results on distinct task splits.
• Figure 4: Upper: The detection accuracy on different $\kappa$. "0" denote w/o adaptive temperature scaling. Lower: SR-CIS performence on 20-split ImageNet-R with different choice of $e$.
• Figure 5: LLaVA generates a description of this image and puts it into the stable diffusion demo to generate the image.