When Graph Neural Networks Meet Dynamic Mode Decomposition

TL;DR

This work illustrates how DMD can estimate a low-rank, finite-dimensional linear operator based on multiple states of the system, effectively approximating potential nonlinear interactions between nodes in the graph and introduces a family of DMD-GNN models that effectively leverage the low-rank eigenfunctions provided by the DMD algorithm.

Abstract

Graph Neural Networks (GNNs) have emerged as fundamental tools for a wide range of prediction tasks on graph-structured data. Recent studies have drawn analogies between GNN feature propagation and diffusion processes, which can be interpreted as dynamical systems. In this paper, we delve deeper into this perspective by connecting the dynamics in GNNs to modern Koopman theory and its numerical method, Dynamic Mode Decomposition (DMD). We illustrate how DMD can estimate a low-rank, finite-dimensional linear operator based on multiple states of the system, effectively approximating potential nonlinear interactions between nodes in the graph. This approach allows us to capture complex dynamics within the graph accurately and efficiently. We theoretically establish a connection between the DMD-estimated operator and the original dynamic operator between system states. Building upon this foundation, we introduce a family of DMD-GNN models that effectively leverage the low-rank eigenfunctions provided by the DMD algorithm. We further discuss the potential of enhancing our approach by incorporating domain-specific constraints such as symmetry into the DMD computation, allowing the corresponding GNN models to respect known physical properties of the underlying system. Our work paves the path for applying advanced dynamical system analysis tools via GNNs. We validate our approach through extensive experiments on various learning tasks, including directed graphs, large-scale graphs, long-range interactions, and spatial-temporal graphs. We also empirically verify that our proposed models can serve as powerful encoders for link prediction tasks. The results demonstrate that our DMD-enhanced GNNs achieve state-of-the-art performance, highlighting the effectiveness of integrating DMD into GNN frameworks.

Figures (6)

• Figure 1: Illustration on how DMD-GNNs are designed.
• Figure 2: Results for hyperparameter sensitivity and comparison between DMD-GNNs and PIDMD-GNNs via directed and undirected graphs.
• Figure 3: Number of feature dimensions and the actual number of ranks selected from the DMD. From the first row ($\xi = 0.85$): Cora, Citeseer, Pubmed; second row ($\xi = 0.7$): Wisconsin, Texas and Cornell.
• Figure 4: Illustration of the block-wise adjacency matrix used in LRGB experiment. The first edge of the second graph is with the index right after the first graph.
• Figure 5: Heat plot of the graph adjacency matrices sourced from the original dataset, i.e., Chickenpox. One can check that although with the same number of eigenvectors, compared to the original graph adjacency, DMD is able to deliver a dense adjacency in a data-driven manner, e.g., historical records.

Theorems & Definitions (3)

• Lemma 1: Informal
• Lemma 2: Formal Version of Lemma \ref{['lem1']}
• proof