Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Transformers with Attentive Federated Aggregation for Time Series Stock Forecasting

Chu Myaet Thwal, Ye Lin Tun, Kitae Kim, Seong-Bae Park, Choong Seon Hong

TL;DR

The paper tackles time series stock forecasting under data scarcity and privacy constraints by integrating Time2Vec temporal embeddings with a transformer encoder in a federated learning framework. It introduces Attentive Federated Learning (FedAtt) to assign attention-based importance to client updates, enabling better aggregation than FedAvg in heterogeneous data settings. Empirical evaluation on Yahoo Finance data across 45 enterprises shows that the proposed FedAtt-powered time series transformer yields superior forecasting accuracy (lower MSE/MAE/MAPE) compared with SOLO and FedAvg, particularly for smaller or more diverse datasets. This approach demonstrates the practical potential of privacy-preserving, collaborative transformers for finance and can be extended to other data-intensive domains such as medicine.

Abstract

Recent innovations in transformers have shown their superior performance in natural language processing (NLP) and computer vision (CV). The ability to capture long-range dependencies and interactions in sequential data has also triggered a great interest in time series modeling, leading to the widespread use of transformers in many time series applications. However, being the most common and crucial application, the adaptation of transformers to time series forecasting has remained limited, with both promising and inconsistent results. In contrast to the challenges in NLP and CV, time series problems not only add the complexity of order or temporal dependence among input sequences but also consider trend, level, and seasonality information that much of this data is valuable for decision making. The conventional training scheme has shown deficiencies regarding model overfitting, data scarcity, and privacy issues when working with transformers for a forecasting task. In this work, we propose attentive federated transformers for time series stock forecasting with better performance while preserving the privacy of participating enterprises. Empirical results on various stock data from the Yahoo! Finance website indicate the superiority of our proposed scheme in dealing with the above challenges and data heterogeneity in federated learning.

Transformers with Attentive Federated Aggregation for Time Series Stock Forecasting

TL;DR

The paper tackles time series stock forecasting under data scarcity and privacy constraints by integrating Time2Vec temporal embeddings with a transformer encoder in a federated learning framework. It introduces Attentive Federated Learning (FedAtt) to assign attention-based importance to client updates, enabling better aggregation than FedAvg in heterogeneous data settings. Empirical evaluation on Yahoo Finance data across 45 enterprises shows that the proposed FedAtt-powered time series transformer yields superior forecasting accuracy (lower MSE/MAE/MAPE) compared with SOLO and FedAvg, particularly for smaller or more diverse datasets. This approach demonstrates the practical potential of privacy-preserving, collaborative transformers for finance and can be extended to other data-intensive domains such as medicine.

Abstract

Recent innovations in transformers have shown their superior performance in natural language processing (NLP) and computer vision (CV). The ability to capture long-range dependencies and interactions in sequential data has also triggered a great interest in time series modeling, leading to the widespread use of transformers in many time series applications. However, being the most common and crucial application, the adaptation of transformers to time series forecasting has remained limited, with both promising and inconsistent results. In contrast to the challenges in NLP and CV, time series problems not only add the complexity of order or temporal dependence among input sequences but also consider trend, level, and seasonality information that much of this data is valuable for decision making. The conventional training scheme has shown deficiencies regarding model overfitting, data scarcity, and privacy issues when working with transformers for a forecasting task. In this work, we propose attentive federated transformers for time series stock forecasting with better performance while preserving the privacy of participating enterprises. Empirical results on various stock data from the Yahoo! Finance website indicate the superiority of our proposed scheme in dealing with the above challenges and data heterogeneity in federated learning.
Paper Structure (14 sections, 3 equations, 6 figures, 1 table)

This paper contains 14 sections, 3 equations, 6 figures, 1 table.

Table of Contents

  1. Introduction
  2. Related Work
  3. Time Series Forecasting
  4. Federated Learning
  5. System Architecture
  6. Time2Vector Embeddings
  7. Transformer Encoder
  8. Attentive Federated Learning
  9. Experiments
  10. Baselines
  11. Datasets
  12. Implementation details
  13. Experiment Results
  14. Conclusion

Figures (6)

  • Figure 1: Visualization of daily stock trend for NVS and DPZ datasets. Values are retrieved from Yahoo! Finance website yahoofinance.
  • Figure 2: An overview of FedAvg and FedAtt schemes.
  • Figure 3: Architecture of a time series forecasting transformer.
  • Figure 4: Visualizing data separation of MSFT and AMZN datasets for time series forecasting. Best viewed in color.
  • Figure 5: Visualization of forecasting results on IBM dataset.
  • ...and 1 more figures