Neural Chains and Discrete Dynamical Systems

TL;DR

The paper investigates neural chains—transformer-like networks without self-attention—as discrete dynamical systems derived from discretized neural integral equations and PDEs. By comparing standard finite-difference discretization with PINN learning for the Burgers and Eikonal equations in 1D, it shows that both approaches converge to essentially the same dynamical knowledge, while PINNs rely on random, high-entropy weight matrices and incur larger parameter counts and training costs, compromising explainability. A key finding is the degeneracy between random PINN weights and structured FD matrices, with random weights often achieving correct solutions even for challenging problems, though with limitations for discontinuous cases. The study emphasizes the potential of PINNs and neural chains for high-dimensional problems, while calling for deeper exploration of stochastic dynamics, non-uniform grids, and mechanistic interpretability in physics-informed AI. Overall, the work provides a framework to understand how physics-based numerical methods and ML-based learning can yield complementary insights for solving PDEs and related discretized systems.

Abstract

We inspect the analogy between machine-learning (ML) applications based on the transformer architecture without self-attention, {\it neural chains} hereafter, and discrete dynamical systems associated with discretised versions of neural integral and partial differential equations (NIE, PDE). A comparative analysis of the numerical solution of the (viscid and inviscid) Burgers and Eikonal equations via standard numerical discretization (also cast in terms of neural chains) and via PINN's learning is presented and commented on. It is found that standard numerical discretization and PINN learning provide two different paths to acquire essentially the same knowledge about the dynamics of the system. PINN learning proceeds through random matrices which bear no direct relation to the highly structured matrices associated with finite-difference (FD) procedures. Random matrices leading to acceptable solutions are far more numerous than the unique tridiagonal form in matrix space, which explains why the PINN search typically lands on the random ensemble. The price is a much larger number of parameters, causing lack of physical transparency (explainability) as well as large training costs with no counterpart in the FD procedure. However, our results refer to one-dimensional dynamic problems, hence they don't rule out the possibility that PINNs and ML in general, may offer better strategies for high-dimensional problems.

Figures (9)

• Figure 1: The basic blocks of the DNN architecture. The procedure is repeated over a very large set of input data.
• Figure 2: Solution to the 1D Burgers' equation is shown at time $t=0.3s$. In the left panel, the solutions obtained by the dynamical model for the 1D viscid Burgers' equation is compared with the PINN solution. It acts as a validation that the dynamical system model well represents the solution process of a trained deep neural network. In the right panel, solutions from two separate runs of the PINN are plotted against the FDM pseudo-spectral solution.
• Figure 3: The weight matrices for layers 1,4 and 7 of the PINN solution of the viscous Burgers' equation in fluid dynamics for two separate, independent runs.
• Figure 4: The figure shows the evolution of a 2D input signal as it passes though the PINN architecture learning the viscous Burgers' equation, across two independent, randomly initialised training runs. The left panel shows the heatmap of the activation values of each neuron in the network. The right panel shows the conversion of a 2-feature input signal $Z[x,t]$ into a 100-feature latent space and eventually a scalar as the output. Only 15 latent features (or neural trajectories) are shown here. Input and output layers are representatively shown in the center instead of their corresponding exact neuron indices.
• Figure 5: This figure represents the inability of the standard PINN to learn the solution to Burger's equation when the viscosity is turned off, i.e., Euler dynamics are followed. Furthermore, the problem becomes more challenging with the introduction of discontinuous initial conditions as in this case given by Eq. \ref{['eq:IC']}.