Towards Efficient Large Scale Spatial-Temporal Time Series Forecasting via Improved Inverted Transformers

TL;DR

EiFormer is introduced, an improved inverted transformer architecture that maintains the adaptive capabilities of iTransformer while reducing computational complexity to linear scale while reducing computational complexity to linear scale.

Abstract

Time series forecasting at scale presents significant challenges for modern prediction systems, particularly when dealing with large sets of synchronized series, such as in a global payment network. In such systems, three key challenges must be overcome for accurate and scalable predictions: 1) emergence of new entities, 2) disappearance of existing entities, and 3) the large number of entities present in the data. The recently proposed Inverted Transformer (iTransformer) architecture has shown promising results by effectively handling variable entities. However, its practical application in large-scale settings is limited by quadratic time and space complexity ($O(N^2)$) with respect to the number of entities $N$. In this paper, we introduce EiFormer, an improved inverted transformer architecture that maintains the adaptive capabilities of iTransformer while reducing computational complexity to linear scale ($O(N)$). Our key innovation lies in restructuring the attention mechanism to eliminate redundant computations without sacrificing model expressiveness. Additionally, we incorporate a random projection mechanism that not only enhances efficiency but also improves prediction accuracy through better feature representation. Extensive experiments on the public LargeST benchmark dataset and a proprietary large-scale time series dataset demonstrate that EiFormer significantly outperforms existing methods in both computational efficiency and forecasting accuracy. Our approach enables practical deployment of transformer-based forecasting in industrial applications where handling time series at scale is essential.

Figures (11)

• Figure 1: Of the 23,849 entities sampled from our network over a two-year period, 14,252 remained active throughout the duration. Meanwhile, 1,974 entities emerged and were not present during the first week of the sampled period. In addition, 5,880 entities ceased activity before the final week of the period.
• Figure 2: Different spatial-temporal modeling designs. (1, 2): patch vs. entity embedding. (3, 4): temporal patch vs. spatial entity attention. (5, 6): feature vs. time MLP. PatchTST nie2022time: (1) + (3) + (5); iTransformer liu2023itransformer: (2) + (4) + (6); TSMixer chen2023tsmixer: (6) + (5).
• Figure 3: The proposed EiFormer architecture. The randomly initialized key matrix of the initial attention layer of each Efficient iTransformer Module is frozen, serving as the random projection operator.
• Figure 4: Design progression of EiFormer.
• Figure 5: Dataset statistics.