Randomness and signal propagation in physics-informed neural networks (PINNs): A neural PDE perspective

TL;DR

The paper investigates why PINN weight matrices appear statistically random and how this randomness shapes signal propagation, by analyzing trained networks for the one-dimensional Burgers’ equation through random matrix theory and a neural PDE perspective. It finds that learned weights lie in a high-entropy regime compatible with the circular law and Marchenko-Pastur law, with outliers more pronounced in the inviscid case, and interprets the network as a discretization of a neural advection-diffusion PDE where numerical stability of the time-stepping scheme governs dynamics. Explicit-layer propagation tends to be numerically unstable, while implicit or higher-order schemes stabilize signal evolution, revealing a fundamental link between discretization stability and network dynamics. These results connect weight statistics to dynamical behavior and offer design principles for stable physics-informed neural networks in physics-informed contexts.

Abstract

Physics-informed neural networks (PINNs) often exhibit weight matrices that appear statistically random after training, yet their implications for signal propagation and stability remain unsatisfactorily understood, let alone the interpretability. In this work, we analyze the spectral and statistical properties of trained PINN weights using viscous and inviscid variants of the one-dimensional Burgers' equation, and show that the learned weights reside in a high-entropy regime consistent with predictions from random matrix theory. To investigate the dynamical consequences of such weight structures, we study the evolution of signal features inside a network through the lens of neural partial differential equations (neural PDEs). We show that random and structured weight matrices can be associated with specific discretizations of neural PDEs, and that the numerical stability of these discretizations governs the stability of signal propagation through the network. In particular, explicit unstable schemes lead to degraded signal evolution, whereas stable implicit and higher-order schemes yield well-behaved dynamics for the same underlying neural PDE. Our results offer an explicit example of how numerical stability and network architecture shape signal propagation in deep networks, in relation to random matrix and neural PDE descriptions in PINNs.

Figures (7)

• Figure 1: The formation of a shock front under viscous and inviscid Burgers dynamics at $t = 0.3$. The left panel shows the PINN predicted solution for viscous variant, compared with the RK-4 pseudo-spectral method for reference. The right panel shows the inviscid case and the failure of PINN to learn the correct dynamics.
• Figure 2: The figure shows KDE-fitted probability density functions (PDFs) and MLE (via negative log-likelihood) for the PINN weight matrices trained on the Burgers equation. Panels correspond to weights from early, middle, and late hidden layers. The generalized Gaussian PDFs are parameterized by mean $\mu$ and standard deviation $\sigma$. Notably for the inviscid case, no generalized Gaussian provides a good fit (see Sec. \ref{['asec:fits']}) for the early and middle layers; the MLE is dominated by the distribution tails, indicating a heavy-tailed structure consistent with implicit self-regularization MartinMahoney2018.
• Figure 3: Spectral analysis of network weight matrices for (a) 1D viscous Burgers’ equation and (b) 1D inviscid Burgers’ equation. Top row: Eigenvalue distributions of a random Gaussian matrix (left) and trained network layers (right), illustrating agreement with the circular law. Bottom left: Singular values of the weight matrices across layers compared with a random Gaussian reference. Bottom right: Empirical density of the squared singular values $(\sigma^2)$ of the normalized covariance matrix $WW^{T}$, compared with the Marchenko-Pastur (MP) law.
• Figure 4: (a) A random Gaussian kernel with diagonal scaling, corresponding to a diagonally dominant weight matrix; (b) The signal evolution over $N_R=5$ independent runs for the diagonally-scaled Random Gaussian kernel. The top panel shows the heat maps of the average activation of each neuron, while the bottom panel shows the trajectories of the signal features through the network. The model has 10 hidden layers, with 16 neurons each.
• Figure 5: The representation of the weight matrix obtained from Eq. \ref{['eq:weights_adv_diff']}, with imposed periodic boundary.