AdaptCL: Adaptive Continual Learning for Tackling Heterogeneity in Sequential Datasets

TL;DR

AdaptCL tackles heterogeneity in sequential, task-agnostic continual learning by combining fine-grained data-driven pruning with task-agnostic parameter isolation. This adaptive approach enables selective parameter reuse and protective freezing across data with varying size, complexity, and similarity, mitigating catastrophic forgetting without manual module selection. Empirical results across MNIST Variants, DomainNet, and food-quality datasets show improved average accuracy and robust forgetting control while using fewer parameters than many baselines. The work demonstrates strong cross-domain applicability and positions adaptive pruning as a practical route for scalable continual learning in heterogeneous environments.

Abstract

Managing heterogeneous datasets that vary in complexity, size, and similarity in continual learning presents a significant challenge. Task-agnostic continual learning is necessary to address this challenge, as datasets with varying similarity pose difficulties in distinguishing task boundaries. Conventional task-agnostic continual learning practices typically rely on rehearsal or regularization techniques. However, rehearsal methods may struggle with varying dataset sizes and regulating the importance of old and new data due to rigid buffer sizes. Meanwhile, regularization methods apply generic constraints to promote generalization but can hinder performance when dealing with dissimilar datasets lacking shared features, necessitating a more adaptive approach. In this paper, we propose AdaptCL, a novel adaptive continual learning method to tackle heterogeneity in sequential datasets. AdaptCL employs fine-grained data-driven pruning to adapt to variations in data complexity and dataset size. It also utilizes task-agnostic parameter isolation to mitigate the impact of varying degrees of catastrophic forgetting caused by differences in data similarity. Through a two-pronged case study approach, we evaluate AdaptCL on both datasets of MNIST Variants and DomainNet, as well as datasets from different domains. The latter include both large-scale, diverse binary-class datasets and few-shot, multi-class datasets. Across all these scenarios, AdaptCL consistently exhibits robust performance, demonstrating its flexibility and general applicability in handling heterogeneous datasets.

Figures (10)

• Figure 1: (a) Traditional parameter isolation methods divide the network into non-interfering modules during inference. However, these methods are limited to task-specific continual learning (aka task incremental learning). They require manual selection of output layers and parameters, resulting in limited generalization and higher parameter usage. (b) AdaptCL achieves task-agnostic parameter isolation by fine-grained data-driven parameter partitioning, enabling high accuracy on heterogeneous datasets without module selection, while also optimizing parameter reuse and saving resources.
• Figure 2: The Adaptive Continual Learning (AdaptCL) training flow. It facilitates adaptive learning via fine-grained data-driven pruning to respond effectively to variations in data complexity and dataset size. Additionally, it enables task-agnostic parameter isolation to ensure optimal model performance on datasets ranging in similarity without requiring manual selection of modules.
• Figure 3: Examples of input images used in the experiments. (a) Large-Scale, Diverse Binary-Class Food Quality Dataset. (b) Few-Shot, Multi-Class Food Quality Dataset. (c) DomainNet comprises datasets with heterogeneous complexity and size. (d) MNIST Variants with heterogeneous similarity.
• Figure 4: Test accuracy comparison of continual learning methods on the Large-Scale, Diverse Binary-Class Food Quality Dataset. Our proposed method, AdaptCL, achieves higher average accuracy while consistently preventing catastrophic forgetting in real-world applications with heterogeneous data, outperforming other methods. (Best viewed in color)
• Figure 5: Results of continual learning methods on the DomainNet that comprises datasets with heterogeneous complexity and size. AdaptCL achieves the best average accuracy and is the most robust to datasets with varied complexity and size. (Best viewed in color)