Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Time-FFM: Towards LM-Empowered Federated Foundation Model for Time Series Forecasting

Qingxiang Liu, Xu Liu, Chenghao Liu, Qingsong Wen, Yuxuan Liang

TL;DR

Time-FFM, a Federated Foundation Model for Time series forecasting by leveraging pretrained LMs is proposed, which begins by transforming time series into the modality of text tokens and proposes a prompt adaption module to bootstrap LMs for time series reasoning.

Abstract

Unlike natural language processing and computer vision, the development of Foundation Models (FMs) for time series forecasting is blocked due to data scarcity. While recent efforts are focused on building such FMs by unlocking the potential of language models (LMs) for time series analysis, dedicated parameters for various downstream forecasting tasks need training, which hinders the common knowledge sharing across domains. Moreover, data owners may hesitate to share the access to local data due to privacy concerns and copyright protection, which makes it impossible to simply construct a FM on cross-domain training instances. To address these issues, we propose Time-FFM, a Federated Foundation Model for Time series forecasting by leveraging pretrained LMs. Specifically, we begin by transforming time series into the modality of text tokens. To bootstrap LMs for time series reasoning, we propose a prompt adaption module to determine domain-customized prompts dynamically instead of artificially. Given the data heterogeneity across domains, we design a personalized federated training strategy by learning global encoders and local prediction heads. Our comprehensive experiments indicate that Time-FFM outperforms state-of-the-arts and promises effective few-shot and zero-shot forecaster.

Time-FFM: Towards LM-Empowered Federated Foundation Model for Time Series Forecasting

TL;DR

Time-FFM, a Federated Foundation Model for Time series forecasting by leveraging pretrained LMs is proposed, which begins by transforming time series into the modality of text tokens and proposes a prompt adaption module to bootstrap LMs for time series reasoning.

Abstract

Unlike natural language processing and computer vision, the development of Foundation Models (FMs) for time series forecasting is blocked due to data scarcity. While recent efforts are focused on building such FMs by unlocking the potential of language models (LMs) for time series analysis, dedicated parameters for various downstream forecasting tasks need training, which hinders the common knowledge sharing across domains. Moreover, data owners may hesitate to share the access to local data due to privacy concerns and copyright protection, which makes it impossible to simply construct a FM on cross-domain training instances. To address these issues, we propose Time-FFM, a Federated Foundation Model for Time series forecasting by leveraging pretrained LMs. Specifically, we begin by transforming time series into the modality of text tokens. To bootstrap LMs for time series reasoning, we propose a prompt adaption module to determine domain-customized prompts dynamically instead of artificially. Given the data heterogeneity across domains, we design a personalized federated training strategy by learning global encoders and local prediction heads. Our comprehensive experiments indicate that Time-FFM outperforms state-of-the-arts and promises effective few-shot and zero-shot forecaster.
Paper Structure (17 sections, 2 equations, 4 figures, 15 tables, 1 algorithm)

This paper contains 17 sections, 2 equations, 4 figures, 15 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. Methodology
  4. Problem Definition
  5. Model Structure
  6. Training Process
  7. Experiments
  8. Main Results
  9. Few-Shot Forecasting
  10. Zero-Shot Forecasting
  11. Model Analysis
  12. Conclusion and Future Work
  13. Experimental Details
  14. Hyperparameter Sensitivity
  15. Full Results
  16. ...and 2 more sections

Figures (4)

  • Figure 1: (a) Specific prediction models are trained for diverse domains. (b) A unified model is trained for cross-domain time series. (c) The current in-modality adaption in FL setting fine-tunes LM for NLP tasks, with all the trained parameters are exchanged between clients and the server. (d) Our proposal investigates how to construct a FM by unlocking the potential of LM for cross-domain time series forecasting in FL paradigm.
  • Figure 2: Overall architecture of Time-FFM. Each round begins with ➀ downloading global parameters of modality alignment and prompt adaption modules. We ➁ conduct modality alignment to generate patch tokens and ➂ adaptively determine prompt tokens. ➃ The two tokens are input into the LM backbone and ➄ the outputs are projected to generate the prediction results. After local optimization, ➅ the updated parameters of modality alignment and prompt adaption modules are uploaded to the server for aggregation.
  • Figure 3: A showcase of prompt adaption.
  • Figure 4: Hyperparameter sensitivity studies on ILI dataset.