Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Benchmarking Catastrophic Forgetting Mitigation Methods in Federated Time Series Forecasting

Khaled Hallak, Oudom Kem

TL;DR

This work targets catastrophic forgetting in federated time-series forecasting by introducing a reproducible benchmarking framework and evaluating CL methods on non-i.i.d. spatiotemporal data. It adapts six continual learning strategies (Replay, EWC, Online EWC, LwF, SI, Naive Fine-Tuning) to a federated forecasting pipeline and measures forgetting, plasticity, and generalization using a task-wise RMSE matrix across 11 seasonal tasks on the Beijing Air Quality dataset. A base model is initialized via FedAvg on a comprehensive base task $\mathcal{T}_0$, after which continual tasks are learned with event-driven CL mechanisms; results reveal that replay-based methods provide the best stability-plasticity trade-off, albeit with memory and privacy considerations, while regularizers offer varied but limited trade-offs. The findings highlight the need for replay-free or memory-efficient approaches, dynamic hyperparameter tuning, and extended tooling to enable robust, privacy-conscious continual federated forecasting in real-world IoT-and-edge scenarios.

Abstract

Catastrophic forgetting (CF) poses a persistent challenge in continual learning (CL), especially within federated learning (FL) environments characterized by non-i.i.d. time series data. While existing research has largely focused on classification tasks in vision domains, the regression-based forecasting setting prevalent in IoT and edge applications remains underexplored. In this paper, we present the first benchmarking framework tailored to investigate CF in federated continual time series forecasting. Using the Beijing Multi-site Air Quality dataset across 12 decentralized clients, we systematically evaluate several CF mitigation strategies, including Replay, Elastic Weight Consolidation, Learning without Forgetting, and Synaptic Intelligence. Key contributions include: (i) introducing a new benchmark for CF in time series FL, (ii) conducting a comprehensive comparative analysis of state-of-the-art methods, and (iii) releasing a reproducible open-source framework. This work provides essential tools and insights for advancing continual learning in federated time-series forecasting systems.

Benchmarking Catastrophic Forgetting Mitigation Methods in Federated Time Series Forecasting

TL;DR

This work targets catastrophic forgetting in federated time-series forecasting by introducing a reproducible benchmarking framework and evaluating CL methods on non-i.i.d. spatiotemporal data. It adapts six continual learning strategies (Replay, EWC, Online EWC, LwF, SI, Naive Fine-Tuning) to a federated forecasting pipeline and measures forgetting, plasticity, and generalization using a task-wise RMSE matrix across 11 seasonal tasks on the Beijing Air Quality dataset. A base model is initialized via FedAvg on a comprehensive base task , after which continual tasks are learned with event-driven CL mechanisms; results reveal that replay-based methods provide the best stability-plasticity trade-off, albeit with memory and privacy considerations, while regularizers offer varied but limited trade-offs. The findings highlight the need for replay-free or memory-efficient approaches, dynamic hyperparameter tuning, and extended tooling to enable robust, privacy-conscious continual federated forecasting in real-world IoT-and-edge scenarios.

Abstract

Catastrophic forgetting (CF) poses a persistent challenge in continual learning (CL), especially within federated learning (FL) environments characterized by non-i.i.d. time series data. While existing research has largely focused on classification tasks in vision domains, the regression-based forecasting setting prevalent in IoT and edge applications remains underexplored. In this paper, we present the first benchmarking framework tailored to investigate CF in federated continual time series forecasting. Using the Beijing Multi-site Air Quality dataset across 12 decentralized clients, we systematically evaluate several CF mitigation strategies, including Replay, Elastic Weight Consolidation, Learning without Forgetting, and Synaptic Intelligence. Key contributions include: (i) introducing a new benchmark for CF in time series FL, (ii) conducting a comprehensive comparative analysis of state-of-the-art methods, and (iii) releasing a reproducible open-source framework. This work provides essential tools and insights for advancing continual learning in federated time-series forecasting systems.
Paper Structure (20 sections, 12 equations, 3 figures, 2 tables)

This paper contains 20 sections, 12 equations, 3 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Problem Formulation
  3. Experimental Setup
  4. Dataset
  5. Continual Task Design
  6. Preprocessing Pipeline
  7. Learning Protocol
  8. Evaluation Metrics
  9. Continual Learning Methods
  10. Naive Fine-Tuning (Original CL)
  11. Replay Buffer (Experience Replay)
  12. Learning without Forgetting
  13. Elastic Weight Consolidation (EWC)
  14. Synaptic Intelligence (SI)
  15. Results and Discussion
  16. ...and 5 more sections

Figures (3)

  • Figure 1: Overview of the FCL framework.
  • Figure 2: Comparison between the static BaseModel and Naive-CL on the Temperature forecasting target across continual tasks ($\mathcal{T}_1$ to $\mathcal{T}_{11})$
  • Figure 3: Effect of replay sample selection ratio on normalized plasticity