VCformer: Variable Correlation Transformer with Inherent Lagged Correlation for Multivariate Time Series Forecasting

TL;DR

VCformer tackles multivariate time series forecasting by addressing two core challenges: lagged cross-variable correlations and non-stationarity. It introduces Variable Correlation Attention (VCA) to compute and aggregate lagged cross-correlations across variates, and Koopman Temporal Detector (KTD) to model non-stationary dynamics via learned Koopman embeddings and an operator $K_{var}$. The combination yields state-of-the-art results on eight real-world datasets, and VCA proves broadly transferable to other Transformer-based models, improving their multivariate correlation modeling. The approach achieves efficiency through FFT-based lag-correlation computation, with a theoretical complexity of $\mathcal{O}(N^2 T \log T)$, making it practical for large-scale MTS forecasting.

Abstract

Multivariate time series (MTS) forecasting has been extensively applied across diverse domains, such as weather prediction and energy consumption. However, current studies still rely on the vanilla point-wise self-attention mechanism to capture cross-variable dependencies, which is inadequate in extracting the intricate cross-correlation implied between variables. To fill this gap, we propose Variable Correlation Transformer (VCformer), which utilizes Variable Correlation Attention (VCA) module to mine the correlations among variables. Specifically, based on the stochastic process theory, VCA calculates and integrates the cross-correlation scores corresponding to different lags between queries and keys, thereby enhancing its ability to uncover multivariate relationships. Additionally, inspired by Koopman dynamics theory, we also develop Koopman Temporal Detector (KTD) to better address the non-stationarity in time series. The two key components enable VCformer to extract both multivariate correlations and temporal dependencies. Our extensive experiments on eight real-world datasets demonstrate the effectiveness of VCformer, achieving top-tier performance compared to other state-of-the-art baseline models. Code is available at this repository: https://github.com/CSyyn/VCformer.

Figures (8)

• Figure 1: Illustration of the dot-product method to obtain correlations with different lags. For example, $lag=3$ shows the similarity between $v_1$ and $v_2$ (points with the same color)
• Figure 2: Overall structure of VCformer, VCA module (a) and KTD module (b)
• Figure 3: A case visualization for multivariate correlation analysis. The upper part is the multivariate correlation of past series and future series. The bottom part is the learned correlation maps in different layers.
• Figure 4: Model efficiency comparison. The running efficiency of eight models on the Exchange (left) and Traffic (right) dataset with the prediction length $H=96$ and the batch size $B=16$.
• Figure 5: Hyperparameter sensitivity with respect to the encoder layer, the Koopman embedding dimension and the projection dimension of variate tokens.