CASA: CNN Autoencoder-based Score Attention for Efficient Multivariate Long-term Time-series Forecasting

TL;DR

This work introduces CASA, a CNN Autoencoder-based Score Attention module that replaces the standard self-attention in Transformer encoders for multivariate long-term time-series forecasting. By employing channel-wise tokenization and a 1D CNN autoencoder as a score network, CASA approximates the traditional $\frac{QK^T}{\sqrt{d_k}}$ operation with linear complexity in the number of variates $N$, input length $L$, and horizon $H$, thus reducing memory and speeding up inference. Across eight real-world datasets, CASA achieves state-of-the-art accuracy while decreasing resource usage (up to 77.7% memory reduction and 44.0% faster inference) and demonstrates strong robustness to varying data lengths and variate counts. The approach is model-agnostic with respect to tokenization and effectively enhances cross-dimensional information capture, offering a practical alternative to conventional self-attention in multivariate LTSF.

Abstract

Multivariate long-term time series forecasting is critical for applications such as weather prediction, and traffic analysis. In addition, the implementation of Transformer variants has improved prediction accuracy. Following these variants, different input data process approaches also enhanced the field, such as tokenization techniques including point-wise, channel-wise, and patch-wise tokenization. However, previous studies still have limitations in time complexity, computational resources, and cross-dimensional interactions. To address these limitations, we introduce a novel CNN Autoencoder-based Score Attention mechanism (CASA), which can be introduced in diverse Transformers model-agnosticically by reducing memory and leading to improvement in model performance. Experiments on eight real-world datasets validate that CASA decreases computational resources by up to 77.7%, accelerates inference by 44.0%, and achieves state-of-the-art performance, ranking first in 87.5% of evaluated metrics.

Figures (14)

• Figure 1: The validation loss of iTransformer, Transformer, PatchTST, and our model on the Traffic dataset is evaluated. Point-wise and patch-wise implemented models exhibit lower performance compared to channel-wise models. However, while the iTransformer model rapidly saturates, CASA demonstrates consistent learning and achieves the lowest loss value.
• Figure 2: (a) point-wise token (b) patch-wise token (c) channel-wise token
• Figure 3: (a) Conventional Self-Attention. (b) Overall architecture of our CASA block. The time-series data is embedded using channel-wise tokenization. The 1D CNN Autoencoder is then used to compute cross-dimensional information. The softmax output and the value are multiplied element-wise. Our CASA places a strong emphasis on capturing essential cross-dimensional information by calculating high-dimensional spatial relationships before compressing the channel information.
• Figure 4: Results of prediction of Ours and baseline models on Weather dataset. (a) CASA (b) iTransformer (c) SOFTS (d) PatchTST
• Figure 5: Experimental results on the ETTm1, Electricity, and Traffic datasets (with 7, 321, and 862 variates, respectively). Our CASA remains robust across varying input and prediction lengths (48 to 720). Unlike PatchTST, which struggles as the number of variates increases, models like iTransformer and SOFTS, which tokenize variates, exhibit stronger performance.

Theorems & Definitions (6)

• Proposition 1
• proof
• Proposition 2
• proof
• proof
• proof