Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

The Ill-Posed Foundations of Physics-Informed Neural Networks and Their Finite-Difference Variants

Andreas Langer

TL;DR

The paper addresses the analytical foundations of physics-informed neural networks (PINNs) in both gradient-based AD-PINN and finite-difference (FD-PINN) formulations, showing that their optimization problems are ill-posed with non-unique minimizers. It develops a unified framework, proves existence of minimizers under mild regularity and that neural networks of finite width can realize these minimizers, and establishes a precise grid-level equivalence between FD-PINN minimizers and discrete finite-difference solutions. The results reveal a structural distinction: FD-PINNs are effectively tied to the underlying discrete scheme on the stencil, ensuring grid-consistent zero-loss solutions when the discrete PDE is unique, whereas AD-PINNs can admit unbounded families of minimizers that diverge from the true PDE solution. Numerical experiments on Poisson, Schrödinger, and Navier–Stokes problems corroborate the theory, showing FD-PINNs’ robustness and explaining the instability often observed in AD-PINNs, with implications for designing more reliable PDE solvers based on neural networks.

Abstract

Physics-informed neural networks based on automatic differentiation (AD-PINNs) and their finite-difference counterparts (FD-PINNs) are widely used for solving partial differential equations (PDEs), yet their analytical properties remain poorly understood. This work provides a unified mathematical foundation for both formulations. Under mild regularity assumptions on the activation function and for sufficiently wide neural networks of depth at least two, we prove that both the AD- and FD-PINN optimization problems are ill-posed: whenever a minimizer exists, there are in fact infinitely many, and uniqueness fails regardless of the choice of collocation points or finite-difference stencil. Nevertheless, we establish two structural properties. First, whenever the underlying PDE or its finite-difference discretization admits a solution, the corresponding AD-PINN or FD-PINN loss also admits a minimizer, realizable by a neural network of finite width. Second, FD-PINNs are tightly coupled to the underlying finite-difference scheme: every FD-PINN minimizer agrees with a finite-difference minimizer on the grid, and in regimes where the discrete PDE solution is unique, all zero-loss FD-PINN minimizers coincide with the discrete PDE solution on the stencil. Numerical experiments illustrate these theoretical insights: FD-PINNs remain stable in representative forward and inverse problems, including settings where AD-PINNs may fail to converge. We also include an inverse problem with noisy data, demonstrating that FD-PINNs retain robustness in this setting as well. Taken together, our results clarify the analytical limitations of AD-PINNs and explain the structural reasons for the more stable behavior observed in FD-PINNs.

The Ill-Posed Foundations of Physics-Informed Neural Networks and Their Finite-Difference Variants

TL;DR

The paper addresses the analytical foundations of physics-informed neural networks (PINNs) in both gradient-based AD-PINN and finite-difference (FD-PINN) formulations, showing that their optimization problems are ill-posed with non-unique minimizers. It develops a unified framework, proves existence of minimizers under mild regularity and that neural networks of finite width can realize these minimizers, and establishes a precise grid-level equivalence between FD-PINN minimizers and discrete finite-difference solutions. The results reveal a structural distinction: FD-PINNs are effectively tied to the underlying discrete scheme on the stencil, ensuring grid-consistent zero-loss solutions when the discrete PDE is unique, whereas AD-PINNs can admit unbounded families of minimizers that diverge from the true PDE solution. Numerical experiments on Poisson, Schrödinger, and Navier–Stokes problems corroborate the theory, showing FD-PINNs’ robustness and explaining the instability often observed in AD-PINNs, with implications for designing more reliable PDE solvers based on neural networks.

Abstract

Physics-informed neural networks based on automatic differentiation (AD-PINNs) and their finite-difference counterparts (FD-PINNs) are widely used for solving partial differential equations (PDEs), yet their analytical properties remain poorly understood. This work provides a unified mathematical foundation for both formulations. Under mild regularity assumptions on the activation function and for sufficiently wide neural networks of depth at least two, we prove that both the AD- and FD-PINN optimization problems are ill-posed: whenever a minimizer exists, there are in fact infinitely many, and uniqueness fails regardless of the choice of collocation points or finite-difference stencil. Nevertheless, we establish two structural properties. First, whenever the underlying PDE or its finite-difference discretization admits a solution, the corresponding AD-PINN or FD-PINN loss also admits a minimizer, realizable by a neural network of finite width. Second, FD-PINNs are tightly coupled to the underlying finite-difference scheme: every FD-PINN minimizer agrees with a finite-difference minimizer on the grid, and in regimes where the discrete PDE solution is unique, all zero-loss FD-PINN minimizers coincide with the discrete PDE solution on the stencil. Numerical experiments illustrate these theoretical insights: FD-PINNs remain stable in representative forward and inverse problems, including settings where AD-PINNs may fail to converge. We also include an inverse problem with noisy data, demonstrating that FD-PINNs retain robustness in this setting as well. Taken together, our results clarify the analytical limitations of AD-PINNs and explain the structural reasons for the more stable behavior observed in FD-PINNs.
Paper Structure (25 sections, 12 theorems, 65 equations, 8 figures)

This paper contains 25 sections, 12 theorems, 65 equations, 8 figures.

Table of Contents

  1. Introduction
  2. Contributions of this work
  3. Preliminaries
  4. Definitions and Notations
  5. Problem
  6. Neural Networks
  7. Differentiability
  8. AD-PINN Framework
  9. Existence of Minimizers
  10. Non-uniqueness of Minimizers
  11. FD-PINN Framework
  12. Existence of Minimizers
  13. Non-Uniqueness of Minimizers
  14. Implications of non-uniqueness: AD-PINNs vs. FD-PINNs
  15. Numerical Experiments
  16. ...and 10 more sections

Key Result

Proposition 3.1

Let $U \subseteq C^{r_\mathcal{F}}(\Omega,\mathbb{R}^c) \cap C^{r_\mathcal{B}}(\overline{\Omega},\mathbb{R}^c)$. Consider finite collocation sets $\Omega^h = \{z_\mathcal{F}^i\}_{i=1}^{N_\mathcal{F}} \subset \Omega$ and $\Gamma^h = \{z_\mathcal{B}^j\}_{j=1}^{N_\mathcal{B}} \subset \Gamma$ with $N_\m

Figures (8)

  • Figure 1: Schematic overview of the four minimization problems associated with the continuous and discrete functionals $\mathcal{J}$ and $\mathcal{J}^h$, posed over the full space $U$ or over the neural network class $\mathcal{H}$, and their relation to the underlying PDE. Double arrows ($\Rightarrow$) indicate logical implications, either holding by definition or proved in \ref{['thm:MinimizerCoincide']}. Single arrows ($\to$) denote the existence-type relations established in \ref{['thm:MinimizerCoincide', 'thm:equivalent:solution']}.
  • Figure 2: Schematic overview of the two minimization problems for the discrete functionals $\mathcal{J}_{\mathrm{FD}}$ and $\mathcal{J}_\theta$ with the relation to the discrete PDE. Double arrows ($\Rightarrow$) indicate logical implications between the statements in the boxes established in \ref{['prop:Equivalence:Solution:FD']}. Single arrows ($\to$) represent the existence-type relations asserted in \ref{['prop:minimizer:equivalence']}, where a minimizer in one setting guarantees the existence of a corresponding minimizer in the other.
  • Figure 3: Solutions of the Poisson problem \ref{['eq:Poisson']} obtained by FDM, FD-PINN and AD-PINN.
  • Figure 4: Solutions of the Schrödinger equation \ref{['eq:Schrodinger']} obtained by the FD-PINN.
  • Figure 5: Solutions of the Schrödinger equation \ref{['eq:Schrodinger']} obtained by the AD-PINN.
  • ...and 3 more figures

Theorems & Definitions (32)

  • Proposition 3.1
  • proof
  • Corollary 3.2
  • proof
  • Theorem 3.3
  • proof
  • Example 3.4
  • Theorem 3.5: Non-uniqueness of AD-PINN minimizers
  • proof
  • Remark 3.6
  • ...and 22 more