Fighter: Unveiling the Graph Convolutional Nature of Transformers in Time Series Modeling

TL;DR

The paper reframes Transformer encoders for time series as dynamic-graph convolutions, where the attention distribution acts as a learnable adjacency matrix $\mathrm{smx}(d^{-1/2}\mathbf{X}^{(l-1)}_{\text{TF}}\mathbf{W}_Q^{(l)}(\mathbf{X}^{(l-1)}_{\text{TF}}\mathbf{W}_K^{(l)})^{\top})$, enabling forward-pass graph-like feature aggregation, while backward-pass updates to value/FFN parameters mimic GCN learning and induce adaptive graph structure via $\mathbf{W}_Q$ and $\mathbf{W}_K$. Building on this, Fighter eliminates redundant projections and introduces multi-hop graph aggregation via $\mathrm{smx}^{[\kappa_{\ell}]}$ and $\mathrm{blkdiag}(\cdot; \kappa_{\ell})$, yielding a compact yet expressive time-series model with explicit, interpretable temporal edges. The authors demonstrate competitive forecasting performance across standard benchmarks (Electricity, Weather) and cross-domain results (AG News), with analyses of how varying $\kappa$ trade-offs local vs. global dependencies and improves interpretability. Overall, the work provides a unified sequence-graph viewpoint that guides efficient, interpretable design of time-series transformers and suggests extensions to other modalities and architectures.

Abstract

Transformers have achieved remarkable success in time series modeling, yet their internal mechanisms remain opaque. This work demystifies the Transformer encoder by establishing its fundamental equivalence to a Graph Convolutional Network (GCN). We show that in the forward pass, the attention distribution matrix serves as a dynamic adjacency matrix, and its composition with subsequent transformations performs computations analogous to graph convolution. Moreover, we demonstrate that in the backward pass, the update dynamics of value and feed-forward projections mirror those of GCN parameters. Building on this unified theoretical reinterpretation, we propose \textbf{Fighter} (Flexible Graph Convolutional Transformer), a streamlined architecture that removes redundant linear projections and incorporates multi-hop graph aggregation. This perspective yields an explicit and interpretable representation of temporal dependencies across different scales, naturally expressed as graph edges. Experiments on standard forecasting benchmarks confirm that Fighter achieves competitive performance while providing clearer mechanistic interpretability of its predictions.

Figures (4)

• Figure 1: Visual comparison of a Transformer encoder and a flexible GCN, illustrating the equivalence between the Transformer’s attention distribution matrix and the GCN’s adjacency matrix in performing feature aggregation during the forward pass.
• Figure 2: Visual illustration of the Fighter architecture, showcasing its streamlined design with multi-hop graph aggregation and dynamic adjacency matrix for efficient time series modeling.
• Figure 3: Time series forecasting performance of Fighter with $\kappa \in [1, 8]$ compared to Transformer on the Weather dataset. Lower MSE/MAE values indicate superior performance.
• Figure 4: Text classification performance of Fighter with $\kappa \in \{2,3,4,6,8\}$ versus Transformer on the AG News dataset, evaluated across epochs 25 to 100. Higher accuracy reflects better performance.

Theorems & Definitions (2)

• Remark 1
• Remark 2