CVTN: Cross Variable and Temporal Integration for Time Series Forecasting

TL;DR

For time series forecasting with Transformers, the paper addresses the dual challenges of underutilizing historical sequences and overfitting during temporal learning. It introduces CVTN, a two-stage framework with a Cross-Variable Encoder ($CVE$) and a Cross-Temporal Encoder ($CTE$) that decouples historical feature mining from prediction-sequence modeling. The approach achieves state-of-the-art performance on eight real-world datasets, using a look-back window of $96$ and prediction horizons up to $720$, with improvements reflected in $MSE$ and $MAE$ metrics. This work presents a practical, generalizable framework that reduces overfitting while enhancing long-term multivariate forecasting, providing a clear pathway for future research.

Abstract

In multivariate time series forecasting, the Transformer architecture encounters two significant challenges: effectively mining features from historical sequences and avoiding overfitting during the learning of temporal dependencies. To tackle these challenges, this paper deconstructs time series forecasting into the learning of historical sequences and prediction sequences, introducing the Cross-Variable and Time Network (CVTN). This unique method divides multivariate time series forecasting into two phases: cross-variable learning for effectively mining fea tures from historical sequences, and cross-time learning to capture the temporal dependencies of prediction sequences. Separating these two phases helps avoid the impact of overfitting in cross-time learning on cross-variable learning. Exten sive experiments on various real-world datasets have confirmed its state-of-the-art (SOTA) performance. CVTN emphasizes three key dimensions in time series fore casting: the short-term and long-term nature of time series (locality and longevity), feature mining from both historical and prediction sequences, and the integration of cross-variable and cross-time learning. This approach not only advances the current state of time series forecasting but also provides a more comprehensive framework for future research in this field.

Figures (3)

• Figure 1: The CVTN architecture is strategically bifurcated into two key components. On the left, CVE adopts a Clientgao2023client architecture, leveraging the Cross-Variable Transformer to effectively delineate dependencies among variables. In contrast, on the right, CTE utilizes a Convolutional Neural Network (CNN) architecture, meticulously designed to decode cross-temporal dependencies.
• Figure 2: Observation of the model's loss trend on the Electricity and Traffic datasets. Training was fixed for 10 epochs with an early stopping tolerance of 3. Training was terminated upon exceeding this tolerance level.
• Figure 3: Comparative Analysis of the Original Transformer versus a Decoder-Only Transformer Model on Electricity and Traffic Datasets. The Decoder-Only model operates with a reduced input of merely 8 historical tokens, distinctly fewer than the 48 tokens utilized in the standard Transformer setup.