Transformer Encoder and Multi-features Time2Vec for Financial Prediction

TL;DR

The paper addresses financial time-series forecasting, where short-term fluctuations and long-term dependencies complicate predictions. It introduces TT2VFin, a neural architecture that fuses Time2Vec temporal encoding with a Transformer Encoder and employs correlation-based feature selection and GMNN feature aggregation to exploit inter-stock relationships. The study shows Time2Vec encoding outperforms traditional positional encoding and that correlation-driven multi-feature inputs improve accuracy, achieving competitive performance with fewer parameters than RNN/LSTM baselines. The approach demonstrates strong generalization and potential applicability to other financial and time-series forecasting problems, including intra-day data.

Abstract

Financial prediction is a complex and challenging task of time series analysis and signal processing, expected to model both short-term fluctuations and long-term temporal dependencies. Transformers have remarkable success mostly in natural language processing using attention mechanism, which also influenced the time series community. The ability to capture both short and long-range dependencies helps to understand the financial market and to recognize price patterns, leading to successful applications of Transformers in stock prediction. Although, the previous research predominantly focuses on individual features and singular predictions, that limits the model's ability to understand broader market trends. In reality, within sectors such as finance and technology, companies belonging to the same industry often exhibit correlated stock price movements. In this paper, we develop a novel neural network architecture by integrating Time2Vec with the Encoder of the Transformer model. Based on the study of different markets, we propose a novel correlation feature selection method. Through a comprehensive fine-tuning of multiple hyperparameters, we conduct a comparative analysis of our results against benchmark models. We conclude that our method outperforms other state-of-the-art encoding methods such as positional encoding, and we also conclude that selecting correlation features enhance the accuracy of predicting multiple stock prices.

Figures (5)

• Figure 1: Correlation analysis of stock indices: auto-correlation and cross-correlation of market trends compared to Exxon Mobil, inspired by Shen2012StockMF.
• Figure 2: The preprocessing pipeline. Steps: Fill and Moving Average, Percentage Change Computing, Normalization, Aggregation using GMNN.
• Figure 3: The postprocessing pipeline. Steps: Denormalization, Reverse Percentage Change, Reverse Moving Average. $\mathbf{x}$ is the original normalized values, $\mathbf{\widehat{x}}$ is the predicted normalized values, $\mathbf{\widehat{y}}$ is the predicted percentage change values, $\mathbf{v}$ is the original moving average values, $\mathbf{\widehat{v}}$ is the predicted moving average values, $\mathbf{z}$ is the original close prices, $\mathbf{\widehat{z}}$ is the predicted close prices.
• Figure 4: The proposed model and architectures. Proposed model (top), Time2Vec module (left mid), Transformer Encoders module (right mid), Encoder Block (bottom). MA stands for Multi-head Attention, LN stands for Layer Normalization. The solid lines are residual connections.
• Figure 5: The predicted close prices. S&P500 predicted close prices and orginal close prices are displayed in the same interval of 6-month (from 2024-08-01 to 2025-02-05) together with its trend and predicted trend. The red line demonstrates the last collected date we did in Section \ref{['sec:data']}. Note that a similar pattern of close prices and trends are attained by the NASDAQ stock.