Class-incremental Learning for Time Series: Benchmark and Evaluation

TL;DR

The paper tackles Time Series Class-incremental Learning (TSCIL) in non-stationary environments where new classes appear over time, risking catastrophic forgetting of earlier classes. It proposes a unified evaluation framework and open-source benchmark to standardize comparisons of generic CIL methods and time-series-specific techniques, including regularization, experience replay, and generative replay, under standard and privacy-sensitive settings. Through comprehensive experiments across HAR and gesture datasets, the authors show that replay-based methods generally outperform regularization approaches, and that LayerNorm significantly improves replay performance, while BatchNorm can hamper it. The work provides a solid baseline and practical insights to guide future TSCIL research and framework development.

Abstract

Real-world environments are inherently non-stationary, frequently introducing new classes over time. This is especially common in time series classification, such as the emergence of new disease classification in healthcare or the addition of new activities in human activity recognition. In such cases, a learning system is required to assimilate novel classes effectively while avoiding catastrophic forgetting of the old ones, which gives rise to the Class-incremental Learning (CIL) problem. However, despite the encouraging progress in the image and language domains, CIL for time series data remains relatively understudied. Existing studies suffer from inconsistent experimental designs, necessitating a comprehensive evaluation and benchmarking of methods across a wide range of datasets. To this end, we first present an overview of the Time Series Class-incremental Learning (TSCIL) problem, highlight its unique challenges, and cover the advanced methodologies. Further, based on standardized settings, we develop a unified experimental framework that supports the rapid development of new algorithms, easy integration of new datasets, and standardization of the evaluation process. Using this framework, we conduct a comprehensive evaluation of various generic and time-series-specific CIL methods in both standard and privacy-sensitive scenarios. Our extensive experiments not only provide a standard baseline to support future research but also shed light on the impact of various design factors such as normalization layers or memory budget thresholds. Codes are available at https://github.com/zqiao11/TSCIL.

Figures (7)

• Figure 1: Schematic of Time Series Class-incremental Learning (TSCIL) process on a dynamic task sequence. Each task introduces new classes (c1 to c6), separated by clear task boundaries. The model undergoes sequential training on the tasks. After training on each task, the model needs to recognize all classes encountered thus far without catastrophic forgetting. The previously learned parameters are adapted for next task's learning.
• Figure 2: Evolution of Average Accuracy ($\mathcal{A}_i$) when using (a) BatchNorm or (b) LayerNorm for normalization. Methods utilizing memory buffer are marked with triangles. Since Offline represents joint training on the entire task sequence, its result shows as a single point instead of a curve.
• Figure 3: Evolution of Final Average Accuracy ($\mathcal{A}_T$) when using different memory budget. The results encompass 4 ER-based methods on 4 datasets, utilizing BatchNrom (top row) or LayerNorm (bottom row) for normalization.
• Figure 4: Ablation study on different types of classifiers. The first two represent the single-head classifier trained with CE and BCE, respectively.
• Figure 5: Intra-class variations in DSA.