SageFormer: Series-Aware Framework for Long-term Multivariate Time Series Forecasting

TL;DR

A novel series-aware framework, explicitly designed to emphasize the significance of interseries dependencies, that seamlessly integrates with existing Transformer-based models, enriching their ability to comprehend interseries relationships.

Abstract

In the burgeoning ecosystem of Internet of Things, multivariate time series (MTS) data has become ubiquitous, highlighting the fundamental role of time series forecasting across numerous applications. The crucial challenge of long-term MTS forecasting requires adept models capable of capturing both intra- and inter-series dependencies. Recent advancements in deep learning, notably Transformers, have shown promise. However, many prevailing methods either marginalize inter-series dependencies or overlook them entirely. To bridge this gap, this paper introduces a novel series-aware framework, explicitly designed to emphasize the significance of such dependencies. At the heart of this framework lies our specific implementation: the SageFormer. As a Series-aware Graph-enhanced Transformer model, SageFormer proficiently discerns and models the intricate relationships between series using graph structures. Beyond capturing diverse temporal patterns, it also curtails redundant information across series. Notably, the series-aware framework seamlessly integrates with existing Transformer-based models, enriching their ability to comprehend inter-series relationships. Extensive experiments on real-world and synthetic datasets validate the superior performance of SageFormer against contemporary state-of-the-art approaches.

Figures (10)

• Figure 1: Illustration of three different ways of modeling series dependencies: (a) The proposed series-aware framework. Prior to the original input tokens into the Transformer encoder, we incorporate learnable global tokens to capture the intrinsic features of each series. The embedding tokens are processed through multiple SageFormer layers, where temporal encoding and graph aggregation are performed iteratively. (b) The series-mixing framework amalgamates all series at once and employs temporal encoding or GNNs to process them. (c) The series-independent framework handles each series separately, improving the learning of unique temporal patterns for different series.
• Figure 2: Illustration of the iterative message-passing process in SageFormer. Each layer begins with graph aggregation, where global tokens from all series are gathered and processed by the multi-hop GNN component (leftmost rectangle). Graph-enhanced global tokens are then dispatched to their original series and encoded by TEB. The weights of each TEB are shared among all series.
• Figure 3: MAE / MSE results on Traffic, Electricity, and Weather datasets.
• Figure 4: Evaluation on hyper-parameter impact. (a) MSE against the length of global tokens on the Traffic dataset. (b) MSE against the graph aggregation depth on the Traffic dataset. (c) MSE against the number of nearest neighbors on the Traffic dataset. (d) MSE against SageFormer encoder layers on the Traffic dataset.
• Figure 5: Visualization of 120 timesteps of (a) the Directed Cycle Graph Dataset and (b) the Low-rank Dataset.