Leveraging Multivariate Long-Term History Representation for Time Series Forecasting

TL;DR

LMHR introduces a Long-term Multivariate History Representation framework to enhance time series forecasting on graphs. It integrates a Long-term History Encoder, a non-parametric Hierarchical Representation Retriever, and a Transformer-based Aggregator to fuse long-term multivariate patterns with short-term spatial-temporal modeling, achieving superior accuracy across multiple real-world datasets and showing interpretability through retrieved representations and attention weights. A graph-structure learning component aligns learned dependencies with retrieved long-term cues, and the framework is compatible with a wide range of STGNN backbones. Overall, LMHR demonstrates meaningful improvements, particularly for horizon-12 forecasts and rapidly changing patterns, offering practical gains and broader applicability in spatial-temporal forecasting tasks.

Abstract

Multivariate Time Series (MTS) forecasting has a wide range of applications in both industry and academia. Recent advances in Spatial-Temporal Graph Neural Network (STGNN) have achieved great progress in modelling spatial-temporal correlations. Limited by computational complexity, most STGNNs for MTS forecasting focus primarily on short-term and local spatial-temporal dependencies. Although some recent methods attempt to incorporate univariate history into modeling, they still overlook crucial long-term spatial-temporal similarities and correlations across MTS, which are essential for accurate forecasting. To fill this gap, we propose a framework called the Long-term Multivariate History Representation (LMHR) Enhanced STGNN for MTS forecasting. Specifically, a Long-term History Encoder (LHEncoder) is adopted to effectively encode the long-term history into segment-level contextual representations and reduce point-level noise. A non-parametric Hierarchical Representation Retriever (HRetriever) is designed to include the spatial information in the long-term spatial-temporal dependency modelling and pick out the most valuable representations with no additional training. A Transformer-based Aggregator (TAggregator) selectively fuses the sparsely retrieved contextual representations based on the ranking positional embedding efficiently. Experimental results demonstrate that LMHR outperforms typical STGNNs by 10.72% on the average prediction horizons and state-of-the-art methods by 4.12% on several real-world datasets. Additionally, it consistently improves prediction accuracy by 9.8% on the top 10% of rapidly changing patterns across the datasets.

Figures (8)

• Figure 1: Traffic flow examples from PEMS04 dataset. (a) Two sensors (0 and 235) show complex and various spatial-temporal correlations and similarities over a long-term history. (b) Similar temporal patterns from sensors 0 and 235 at different times, showing potential value from long-term multivariate history. (c) Different temporal patterns (magnitude and trend) from sensors 0 and 235 in a short-term window, demonstrating the unreliability of current methods that rely solely on short-term multivariate history.
• Figure 2: The overview of the proposed LMHR framework for MTS forecasting. The yellow, red, green, and blue curves represent long-term time series information from the node itself, from other nodes, short-term information from MTS, and forecasting-related information, respectively. Long-term histories of multivariate MTS series are separately encoded by the LHEncoder. HRetriever hierarchically searches potential useful information and outputs the top-$k$ similar representations from all nodes to TAggregator. TAggregator fuses the retrieved segments' contextual representations with corresponding ranking position embeddings. LMHR also leverages the representations from HRetriever to generate an adjacency matrix used in STGNN to extract the short-term dependencies. The representations from LHEncoder, HRetriever and TAggregator are fused together in final forecasting.
• Figure 3: The Hierarchical Representation Retriever. The colors yellow, red and green represent long-term information from the node itself, from other nodes, and the short-term segment-level representation nearest the prediction, respectively. HRetriever first finds top $K_n$ similar series through series-level retrieval and constructs the adjacency matrix, then it picks out the top $K_s$ similar segment representations from the retrieved similar series using segment-level retrieval.
• Figure 4: The Transformer-based Aggregator. The colors yellow, red and green in Fig. \ref{['fig:aggregator']} have the same meaning as those in Fig. \ref{['fig:retriever']}. The blue represents the embedding or representation of the prediction segment, and the grey is the ranking position embedding of the retrieved segment. The TAggregator utilizes positional embedding of the target segment to absorb retrieved segment representations from MTS long-term history using the Transformer method. Learnable weights are dynamically updated during training.
• Figure 5: Retrieved similar sensors from series-level of HRetriever. The top 5 retrieved sensors have similar long-term patterns compared with sensor 0.