T-Graphormer: Using Transformers for Spatiotemporal Forecasting

TL;DR

This work introduces T-Graphormer, a Transformer-based framework that extends Graphormer with temporal encodings to model spatiotemporal dependencies jointly, mitigating the need for separate spatial and temporal modules. By incorporating centrality, learned temporal positions, and SPD-based spatial biases directly into the attention mechanism, it learns rich spacetime patterns from traffic graphs. Empirical results on standard traffic datasets show state-of-the-art performance with substantial RMSE/MAPE gains, and ablations highlight the critical role of positional and spatial encodings as well as special tokens. The approach offers a scalable, unified perspective on spatiotemporal forecasting, with potential extensions to dynamic graphs and more scalable attention mechanisms for large networks.

Abstract

Spatiotemporal data is ubiquitous, and forecasting it has important applications in many domains. However, its complex cross-component dependencies and non-linear temporal dynamics can be challenging for traditional techniques. Existing methods address this by learning the two dimensions separately. Here, we introduce Temporal Graphormer (T-Graphormer), a Transformer-based approach capable of modelling spatiotemporal correlations simultaneously. By adding temporal encodings in the Graphormer architecture, each node attends to all other tokens within the graph sequence, enabling the model to learn rich spacetime patterns with minimal predefined inductive biases. We show the effectiveness of T-Graphormer on real-world traffic prediction benchmark datasets. Compared to state-of-the-art methods, T-Graphormer reduces root mean squared error (RMSE) and mean absolute percentage error (MAPE) by up to 20% and 10%.

Figures (12)

• Figure 1: Visualization of a multivariate time series with graph structure. In this example, the graph remains static over time, similar to traffic networks. The node colours become more opaque to indicate more recent data points.
• Figure 2: Information flow differences between existing methods and T-Graphormer. In part (a), the double arrows represent the cross-sectional input for each learning module, and the circle represents the fusion between the modules. For instance, GraphWaveNet has temporal modules (CNN) that treat the "concatenated" node features as the hidden embeddings ($\bm{t_i}$), then its spatial learning modules (GNN) "concatenate" the time features to produce node embeddings ($\bm{v_i}$). Such information flow is similar in attention-based models (e.g. STAEformer, PDFormer). The single arrows demonstrate that vector $\bm{t_3}$ is updated based on $\bm{t_1}\text{-}\bm{t_2}$, while $\bm{v_5}$ is updated based on $\bm{v_1}\text{-}\bm{v_4}$. In comparison, in part (b), each token in T-Graphormer attends to all other tokens.
• Figure 3: T-Graphormer model architecture. Centrality encoding and positional encoding are added to the node feature vector, which is then passed into the Graphormer encoder blocks. Edge weights are used to compute spatial encodings that determine the attention bias. Finally, the prediction layer maps the learned representation onto the output space. A cls token (white fill) is added to the beginning of the sequence. The illustration is consistent with the example in Figure \ref{['fig:time_series']}.
• Figure 4: Attention heatmap averaged across time and space shows T-Graphormer learns spatial and temporal relationships. On the PEMS-BAY dataset, node-node attention scores reveal strong self-attention, symmetric clusters in the bottom right corner, and selectivity to important nodes (shown in columns with 0.5 values). Time-time attention scores reveal strong emphases on recent time points. It also shows temporal grouping, where earlier and later time points attend within each other but not between (shown by the colour gradient in the top-left and bottom-right corners). This pattern is more obvious in the METR-LA dataset in Figure \ref{['fig:metr-attn']}.
• Figure 5: Ablation results on the METR-LA dataset with a forecasting horizon of 12. Ablation trends are similar for MAPE (Figure \ref{['fig: ablation_supp']}). Left: Structural encoding ablations. Right: Special token ablations.