Learning Dynamic Representations via An Optimally-Weighted Maximum Mean Discrepancy Optimization Framework for Continual Learning

TL;DR

This paper tackles catastrophic forgetting in continual learning by proposing Optimally-Weighted Maximum Mean Discrepancy (OWMMD), which combines a Multi-Level Feature Matching Mechanism (MLFMM) with Adaptive Regularization Optimization (ARO) to regulate intermediate representations across tasks. The approach leverages a kernel-based Maximum Mean Discrepancy (MMD) regularization across layerwise features and learns per-layer weights via softmax normalization to balance retention and adaptation. Empirical results on CIFAR-10/100 and Tiny-ImageNet under Task-IL and Class-IL settings show state-of-the-art performance with reduced backward transfer and robust behavior on complex datasets. The findings demonstrate that integrating representation-level regularization with adaptive, task-aware weighting and memory-based rehearsal yields a scalable and effective solution for continual learning, with potential extensions to larger-scale data and transformer-based architectures.

Abstract

Continual learning has emerged as a pivotal area of research, primarily due to its advantageous characteristic that allows models to persistently acquire and retain information. However, catastrophic forgetting can severely impair model performance. In this study, we address network forgetting by introducing a novel framework termed Optimally-Weighted Maximum Mean Discrepancy (OWMMD), which imposes penalties on representation alterations via a Multi-Level Feature Matching Mechanism (MLFMM). Furthermore, we propose an Adaptive Regularization Optimization (ARO) strategy to refine the adaptive weight vectors, which autonomously assess the significance of each feature layer throughout the optimization process, The proposed ARO approach can relieve the over-regularization problem and promote the future task learning. We conduct a comprehensive series of experiments, benchmarking our proposed method against several established baselines. The empirical findings indicate that our approach achieves state-of-the-art performance.

Figures (6)

• Figure 1: The learning process of the proposed framework during the first task, where the model learns data samples without adaptive regularization. Images of Current Task Dataset and their logits are stored into the buffer via reservoir sampling for future task regularization. Different colored arrows indicate distinct data paths.
• Figure 2: The learning process of the proposed framework during the subsequent tasks, where the adaptive regularization is applied to relieve network forgetting. The regularization term leverages layer-wise Maximum Mean Discrepancy (MMD) between the intermediate features of the teacher network (previous task model) and the student network (current model), guided by adaptive weights to prioritize critical layers. Current task data are processed to generate logits, which are then stored into the buffer via reservoir sampling, ensuring balanced retention of historical knowledge while adapting to new tasks. In the figure, Different colored arrows indicate distinct data paths.
• Figure 3: Forgetting curve analysis of various models on CIFAR-10 , CIFAR-100 and Tiny-ImageNet datasets under class-incremental and task-incremental learning settings.
• Figure 4: Dynamic Adjustment Process of Layer-wise Adaptive Weights Across Tasks.
• Figure 5: The distribution of the final adaptive weights after learning all tasks.