Stretched and measured neural predictions of complex network dynamics

TL;DR

The work tackles learning dynamical systems on graphs by framing the problem as vector-field learning and addressing generalization beyond i.i.d. assumptions. It introduces a graph neural network that splits dynamics into self- and neighbor-interactions with quadratic terms, trained via a robust $\ell^1$-based loss augmented by a variance penalty. A neural network null model and a $d$-statistic significance test quantify predictive confidence under distributional shift, enabling reliable inference beyond standard SLT. Across multiple complex dynamics and graph scenarios, including time-series data and irregular sampling, the approach achieves accurate approximation, prediction, and long-range forecasting while providing principled confidence assessments, suggesting practical applicability to diverse complex systems.

Abstract

Differential equations are a ubiquitous tool to study dynamics, ranging from physical systems to complex systems, where a large number of agents interact through a graph with non-trivial topological features. Data-driven approximations of differential equations present a promising alternative to traditional methods for uncovering a model of dynamical systems, especially in complex systems that lack explicit first principles. A recently employed machine learning tool for studying dynamics is neural networks, which can be used for data-driven solution finding or discovery of differential equations. Specifically for the latter task, however, deploying deep learning models in unfamiliar settings - such as predicting dynamics in unobserved state space regions or on novel graphs - can lead to spurious results. Focusing on complex systems whose dynamics are described with a system of first-order differential equations coupled through a graph, we show that extending the model's generalizability beyond traditional statistical learning theory limits is feasible. However, achieving this advanced level of generalization requires neural network models to conform to fundamental assumptions about the dynamical model. Additionally, we propose a statistical significance test to assess prediction quality during inference, enabling the identification of a neural network's confidence level in its predictions.

Figures (7)

• Figure 1: Reconstruction of a two-dimensional vector field representing mass-action kinetics (MAK) i.e., protein-protein interaction dynamics Barzel2013UniversalityDynamicsvoit2000computational (see Tab. \ref{['tab1']} for definition) a) utilizing a feed-forward neural network, and b) utilizing a dedicated graph neural network that conforms to inductive biases, appropriate for modeling dynamics described as a system of coupled ordinary differential equations. The application of these biases extends the domain of vector-field approximation for MAK dynamics well beyond the range of our training data, indicated as black crosses.
• Figure 2: An ensemble of 10 overparameterized feed forward neural networks $\{ \Psi_m(x) \}$ trained independently to approximate $\mathcal{F}(x)=\cos{2x}$ within the range $[-2,2]$ (50 training samples are indicated with black circles). a) Predictions of the models outside the training range, $x\in[-5,5]$. b) Sample variance across ensemble of neural networks $\{ \Psi_m(x) \}$.
• Figure 3: The ratio $R_{a,b}$ between prediction error and the approximation error shows robust generalisation for small changes in distribution parameters. The prediction error is a sample mean obtained when test data $\mathcal{D}:\{\mathbf{x}\sim \mathcal{B}(a,b),\pmb{\mathcal{F}}(\mathbf{x})\}$, the approximation error is a sample mean obtained using $\mathcal{D}_{\text{train}}$.
• Figure 4: Prediction error when the train graph is replaced by a novel graph, as a function of $\textbf{a)}$ the density of the novel graph $\mathcal{G}$, $\textbf{p}$, and $\textbf{b)}$ the size of the novel graph, $|\mathcal{G}|$. The neural networks were tested using samples from uniform distribution and $10$ different ER graphs with parameters $p,n$. The lack of correlation between the error and the size or a density of a graph suggests that neural network models of dynamics can be repurposed to form predictions on novel graphs not observed during training.
• Figure 5: Forecasts of five complex network dynamics are accurate both on a graph used during training, as well as on novel input graphs. The $d$-statistic successfully identifies the limit of model's generalization. a) compares the training loss (green triangle) with the test loss (purple circles), as a function of $\Delta$, where the initial value $\mathbf{x}(t_0)\sim\mathcal{U}(0,1)+\Delta$. The orange circles indicate the fraction of rejected data points based on the $d$-statistic for a given $\Delta$. Here the test graph is isomorphic to the training graph. In $\textbf{b)}$, we changed the test graph to a larger one, $n=15,p=0.3$ and contrast the true dynamics (solid lines) and the neural network forecasts (dotted lines) for a new set of initial values that are sampled from the training distribution. The insets contrast the cumulative distribution of the $d$-statistic in training data (purple) and in test data (orange) and show the distributions in the range up to a critical value for the significance level of $5\%$. The title reports the percentage of accepted data points, i.e. the percentage of datapoints for which the corresponding $d$-statistic fell within the 95% of the null (purple) distribution.