Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

CLeAN: Continual Learning Adaptive Normalization in Dynamic Environments

Isabella Marasco, Davide Evangelista, Elena Loli Piccolomini, Michele Colajanni

Abstract

Artificial intelligence systems predominantly rely on static data distributions, making them ineffective in dynamic real-world environments, such as cybersecurity, autonomous transportation, or finance, where data shifts frequently. Continual learning offers a potential solution by enabling models to learn from sequential data while retaining prior knowledge. However, a critical and underexplored issue in this domain is data normalization. Conventional normalization methods, such as min-max scaling, presuppose access to the entire dataset, which is incongruent with the sequential nature of continual learning. In this paper we introduce Continual Learning Adaptive Normalization (CLeAN), a novel adaptive normalization technique designed for continual learning in tabular data. CLeAN involves the estimation of global feature scales using learnable parameters that are updated via an Exponential Moving Average (EMA) module, enabling the model to adapt to evolving data distributions. Through comprehensive evaluations on two datasets and various continual learning strategies, including Resevoir Experience Replay, A-GEM, and EwC we demonstrate that CLeAN not only improves model performance on new data but also mitigates catastrophic forgetting. The findings underscore the importance of adaptive normalization in enhancing the stability and effectiveness of tabular data, offering a novel perspective on the use of normalization to preserve knowledge in dynamic learning environments.

CLeAN: Continual Learning Adaptive Normalization in Dynamic Environments

Abstract

Artificial intelligence systems predominantly rely on static data distributions, making them ineffective in dynamic real-world environments, such as cybersecurity, autonomous transportation, or finance, where data shifts frequently. Continual learning offers a potential solution by enabling models to learn from sequential data while retaining prior knowledge. However, a critical and underexplored issue in this domain is data normalization. Conventional normalization methods, such as min-max scaling, presuppose access to the entire dataset, which is incongruent with the sequential nature of continual learning. In this paper we introduce Continual Learning Adaptive Normalization (CLeAN), a novel adaptive normalization technique designed for continual learning in tabular data. CLeAN involves the estimation of global feature scales using learnable parameters that are updated via an Exponential Moving Average (EMA) module, enabling the model to adapt to evolving data distributions. Through comprehensive evaluations on two datasets and various continual learning strategies, including Resevoir Experience Replay, A-GEM, and EwC we demonstrate that CLeAN not only improves model performance on new data but also mitigates catastrophic forgetting. The findings underscore the importance of adaptive normalization in enhancing the stability and effectiveness of tabular data, offering a novel perspective on the use of normalization to preserve knowledge in dynamic learning environments.
Paper Structure (14 sections, 20 equations, 3 figures, 2 tables, 1 algorithm)

This paper contains 14 sections, 20 equations, 3 figures, 2 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. Methodology
  4. Notations
  5. Continual Learning
  6. Catastrophic forgetting
  7. Normalization
  8. Our proposal: CLeAN
  9. Experimental results
  10. Datasets
  11. Experimental setup
  12. Metrics
  13. Results
  14. Conclusions

Figures (3)

  • Figure 1: A graphical representation of our proposed normalization scheme: CLeAN.
  • Figure 2: Performance comparison on the UNSW-NB15 dataset. (a–d) Average Accuracy, (e–h) Average AUROC, and (i–l) Average Forgetting are shown across the five daily experiences for different buffer strategies: (a, e, i) Finetuning, (b, f, j) Reservoir, (c, g, k) A-GEM, and (d, h, l) EwC. Each plot compares the results obtained with different normalization techniques (Global, Local, CN, CLeAN).
  • Figure 3: Performance comparison on the CICIDS-2017 dataset. (a-d) Average Accuracy, (e-h) Average AUROC, and (h-l) Average Forgetting are shown across the five daily experiences for different buffer strategies: (a, e, i) Finetuning, (b, f, j) Reservoir, (c, g, k) A-GEM, and (d, h, l) EwC. Each plot compares the results obtained with different normalization techniques (Global, Local, CN, CLeAN).