Evaluating Temporal Plasticity in Foundation Time Series Models for Incremental Fine-tuning

TL;DR

The paper investigates how time series foundation models adapt under continual learning and distribution shifts, introducing a continual-learning evaluation pipeline and metrics to quantify temporal plasticity. It contrasts traditional incremental training with full retraining and zero-shot/few-shot baselines across Time-MoE and Chronos, revealing that foundation models better mitigate plasticity loss and catastrophic forgetting than smaller models. The results show that while incremental fine-tuning improves foundation models, full retraining can still outperform it under substantial shifts, highlighting the need for robust adaptation strategies. The work concludes that scaling pretraining data and refining fine-tuning approaches may be more impactful than building domain-specific small models, and provides a practical framework for assessing temporal plasticity in real-world settings.

Abstract

Time series foundation models excel at diverse time series forecasting tasks, but their capacity for continuous improvement through incremental learning remains unexplored. We present the first comprehensive study investigating these models' temporal plasticity - their ability to progressively enhance performance through continual learning while maintaining existing capabilities. Through experiments on real-world datasets exhibiting distribution shifts, we evaluate both conventional deep learning models and foundation models using a novel continual learning framework. Our findings reveal that while traditional models struggle with performance deterioration during incremental fine-tuning, foundation models like Time-MoE and Chronos demonstrate sustained improvement in predictive accuracy. This suggests that optimizing foundation model fine-tuning strategies may be more valuable than developing domain-specific small models. Our research introduces new evaluation methodologies and insights for developing foundation time series models with robust continuous learning capabilities.

Figures (7)

• Figure 1: Challenges in Time Series Modeling. The figure illustrates model failure caused by temporal distribution shifts, such as the COVID-19 outbreak, and the divergence between historical and forecast data distributions.
• Figure 2: Data distribution shifts in the Flight and CD-Bike datasets.
• Figure 3: Incremental Fine-Tuning Algorithm results for the Flight (top) and CD-Bike (bottom) datasets. Solid lines represent foundation models' fine-tuning performance, dashed lines show small models' fine-tuning results, and dotted lines indicate foundation models' zero-shot performance.
• Figure 4: Full Training Algorithm results for the Flight (top) and CD-Bike (bottom) datasets. Solid lines represent foundation models' full training performance, dashed lines show small models' performance, and dotted lines indicate foundation models' zero-shot results.
• Figure 5: $R^{\text{zero}}_p$ for the Flight (top) and CD-Bike (bottom) datasets.