Understanding the Difficulty of Solving Cauchy Problems with PINNs

TL;DR

The paper investigates why Physics-Informed Neural Networks struggle to solve Cauchy problems, identifying two core issues: the use of $L^2$ residual losses that may not reflect PDE dynamics and an intrinsic approximation gap that prevents neural nets from representing discontinuous or high-frequency solutions. By analyzing first- and second-order equations, it shows that zero training loss does not guarantee the true solution due to characteristics, non-locality, and compact-domain formulation, and it proves that the image set of neural nets may be non-compact, allowing global minima at infinity and machine-precision-limited accuracy. The authors provide mathematical arguments and numerical experiments (e.g., Burgers' equation) to illustrate how these factors lead to persistent errors despite low training loss, and they derive a lower bound on achievable error tied to grid size and machine precision. The study underscores the need for loss designs that respect evolutionary dynamics and for methods capable of representing discontinuities, with implications for improving PINN-based scientific computing. The practical impact is a call for more nuanced training objectives and architectures when applying PINNs to evolution equations in physics and engineering.

Abstract

Physics-Informed Neural Networks (PINNs) have gained popularity in scientific computing in recent years. However, they often fail to achieve the same level of accuracy as classical methods in solving differential equations. In this paper, we identify two sources of this issue in the case of Cauchy problems: the use of $L^2$ residuals as objective functions and the approximation gap of neural networks. We show that minimizing the sum of $L^2$ residual and initial condition error is not sufficient to guarantee the true solution, as this loss function does not capture the underlying dynamics. Additionally, neural networks are not capable of capturing singularities in the solutions due to the non-compactness of their image sets. This, in turn, influences the existence of global minima and the regularity of the network. We demonstrate that when the global minimum does not exist, machine precision becomes the predominant source of achievable error in practice. We also present numerical experiments in support of our theoretical claims.

Figures (8)

• Figure 1: An example where PINNs fail to solve (left), compared to the true solution (right).
• Figure 2: The shaded region is not covered by valid characteristics.
• Figure 3: Neural networks can converge to step functions.
• Figure 4: The effective region $\mathcal{E}(w, b)$.
• Figure 5: The minimal achievable error is determined by $|\log \epsilon|$.

Theorems & Definitions (11)

• theorem 1
• proof
• theorem 2
• proof
• corollary 1
• theorem 3
• proposition 1
• theorem 4
• proof
• theorem 5