Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

On The Temporal Domain of Differential Equation Inspired Graph Neural Networks

Moshe Eliasof, Eldad Haber, Eran Treister, Carola-Bibiane Schönlieb

TL;DR

The paper addresses limitations of fixed first- and second-order temporal dynamics in differential equation–inspired graph neural networks (DE-GNNs) by introducing Time-Dependent DE-GNNs (TDE-GNNs), which learn the temporal order and dependencies from data. It formulates a higher-order ODE, ∑_{p=1}^{o} c_p( H_o^{(l)} ) ∂^p F/∂t^p = s(F(t); G), discretized via forward Euler to yield F^{(l+1)} = ∑_{p=1}^{o} c_p( H_o^{(l)} ) F^{(l-p+1)} + h s(F^{(l)}; G), with history H_o^{(l)} and coefficients c constrained to sum to 1 for stability. The framework supports direct and attention-based parameterizations of c, enabling flexible, data-driven higher-order dynamics, and includes initialization and stability results. Experiments on node classification and spatio-temporal forecasting show that higher-order TDE-GNNs, especially the attention-based variant, outperform existing DE-GNNs and temporal GNNs, particularly on non-homophilic graphs, while providing interpretable learned temporal behavior.

Abstract

Graph Neural Networks (GNNs) have demonstrated remarkable success in modeling complex relationships in graph-structured data. A recent innovation in this field is the family of Differential Equation-Inspired Graph Neural Networks (DE-GNNs), which leverage principles from continuous dynamical systems to model information flow on graphs with built-in properties such as feature smoothing or preservation. However, existing DE-GNNs rely on first or second-order temporal dependencies. In this paper, we propose a neural extension to those pre-defined temporal dependencies. We show that our model, called TDE-GNN, can capture a wide range of temporal dynamics that go beyond typical first or second-order methods, and provide use cases where existing temporal models are challenged. We demonstrate the benefit of learning the temporal dependencies using our method rather than using pre-defined temporal dynamics on several graph benchmarks.

On The Temporal Domain of Differential Equation Inspired Graph Neural Networks

TL;DR

The paper addresses limitations of fixed first- and second-order temporal dynamics in differential equation–inspired graph neural networks (DE-GNNs) by introducing Time-Dependent DE-GNNs (TDE-GNNs), which learn the temporal order and dependencies from data. It formulates a higher-order ODE, ∑_{p=1}^{o} c_p( H_o^{(l)} ) ∂^p F/∂t^p = s(F(t); G), discretized via forward Euler to yield F^{(l+1)} = ∑_{p=1}^{o} c_p( H_o^{(l)} ) F^{(l-p+1)} + h s(F^{(l)}; G), with history H_o^{(l)} and coefficients c constrained to sum to 1 for stability. The framework supports direct and attention-based parameterizations of c, enabling flexible, data-driven higher-order dynamics, and includes initialization and stability results. Experiments on node classification and spatio-temporal forecasting show that higher-order TDE-GNNs, especially the attention-based variant, outperform existing DE-GNNs and temporal GNNs, particularly on non-homophilic graphs, while providing interpretable learned temporal behavior.

Abstract

Graph Neural Networks (GNNs) have demonstrated remarkable success in modeling complex relationships in graph-structured data. A recent innovation in this field is the family of Differential Equation-Inspired Graph Neural Networks (DE-GNNs), which leverage principles from continuous dynamical systems to model information flow on graphs with built-in properties such as feature smoothing or preservation. However, existing DE-GNNs rely on first or second-order temporal dependencies. In this paper, we propose a neural extension to those pre-defined temporal dependencies. We show that our model, called TDE-GNN, can capture a wide range of temporal dynamics that go beyond typical first or second-order methods, and provide use cases where existing temporal models are challenged. We demonstrate the benefit of learning the temporal dependencies using our method rather than using pre-defined temporal dynamics on several graph benchmarks.
Paper Structure (24 sections, 1 theorem, 21 equations, 6 figures, 9 tables, 2 algorithms)

This paper contains 24 sections, 1 theorem, 21 equations, 6 figures, 9 tables, 2 algorithms.

Table of Contents

  1. Introduction
  2. Related Work
  3. Mathematical Background and Motivation
  4. Differential Equations Inspired GNNs (DE-GNNs).
  5. Time Dependent Differential Equations Inspired GNNs
  6. TDE-GNN Learns Higher-Order DEs
  7. Implementing ${\bf c}(\mathcal{H}_{o}^{(l)})$
  8. Experiments
  9. Node Classification
  10. Spatio-Temporal Forecasting
  11. Ablation Study
  12. Conclusions
  13. Proof to Theorem 1
  14. Proof:
  15. Implementation Details
  16. ...and 9 more sections

Key Result

Theorem 1

(Stability of TDE-GNN). For the discretization of eq:TDE_TimeDisc, there exists a vector ${\bf c} = [c_1, \ldots, c_{o}]$ such that the discrete solution is stable.

Figures (6)

  • Figure 1: An illustration of a pendulum.
  • Figure 2: The pendulum location prediction example. (a). Pendulum $y_1(t)$ coordinate vs. time (b) The prediction performance of naive, 1st, and 2nd order models. Higher-order models offer improved predictions.
  • Figure 3: The embedding of input features ${\bf I}$ using an MLPs ($e_1,e_2,e_3$) to obtain $o=3$ initial conditions, followed by our TDE-GNN for stationary problems.
  • Figure 4: The initialization of TDE-GNN for spatio-temporal data with a history of $o=3$.
  • Figure 5: The impact of the model order $o$ on the performance of TDE-GNN.
  • ...and 1 more figures

Theorems & Definitions (2)

  • Theorem 1
  • Remark 1