Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

FP64 is All You Need: Rethinking Failure Modes in Physics-Informed Neural Networks

Chenhui Xu, Dancheng Liu, Amir Nassereldine, Jinjun Xiong

TL;DR

The paper reframes PINN failure modes as precision-induced stalls rather than true local optima, showing that insufficient arithmetic precision (FP32) causes premature stopping of the L-BFGS optimizer and results in apparent failure modes. By switching to FP64, vanilla PINNs reliably solve four PDE benchmarks, revealing a three-phase training dynamics (un-converged, failure, success) whose boundaries shift with precision. The authors demonstrate that the loss and error landscapes remain connected within the same basin, argue for the same-basin hypothesis, and position double-precision training as essential for dependable physics-informed neural PDE solvers. This work highlights arithmetic precision as a critical hyper-parameter in scientific ML, with practical implications for reliability and hardware considerations in PINN deployments.

Abstract

Physics Informed Neural Networks (PINNs) often exhibit failure modes in which the PDE residual loss converges while the solution error stays large, a phenomenon traditionally blamed on local optima separated from the true solution by steep loss barriers. We challenge this understanding by demonstrate that the real culprit is insufficient arithmetic precision: with standard FP32, the LBFGS optimizer prematurely satisfies its convergence test, freezing the network in a spurious failure phase. Simply upgrading to FP64 rescues optimization, enabling vanilla PINNs to solve PDEs without any failure modes. These results reframe PINN failure modes as precision induced stalls rather than inescapable local minima and expose a three stage training dynamic unconverged, failure, success whose boundaries shift with numerical precision. Our findings emphasize that rigorous arithmetic precision is the key to dependable PDE solving with neural networks.

FP64 is All You Need: Rethinking Failure Modes in Physics-Informed Neural Networks

TL;DR

The paper reframes PINN failure modes as precision-induced stalls rather than true local optima, showing that insufficient arithmetic precision (FP32) causes premature stopping of the L-BFGS optimizer and results in apparent failure modes. By switching to FP64, vanilla PINNs reliably solve four PDE benchmarks, revealing a three-phase training dynamics (un-converged, failure, success) whose boundaries shift with precision. The authors demonstrate that the loss and error landscapes remain connected within the same basin, argue for the same-basin hypothesis, and position double-precision training as essential for dependable physics-informed neural PDE solvers. This work highlights arithmetic precision as a critical hyper-parameter in scientific ML, with practical implications for reliability and hardware considerations in PINN deployments.

Abstract

Physics Informed Neural Networks (PINNs) often exhibit failure modes in which the PDE residual loss converges while the solution error stays large, a phenomenon traditionally blamed on local optima separated from the true solution by steep loss barriers. We challenge this understanding by demonstrate that the real culprit is insufficient arithmetic precision: with standard FP32, the LBFGS optimizer prematurely satisfies its convergence test, freezing the network in a spurious failure phase. Simply upgrading to FP64 rescues optimization, enabling vanilla PINNs to solve PDEs without any failure modes. These results reframe PINN failure modes as precision induced stalls rather than inescapable local minima and expose a three stage training dynamic unconverged, failure, success whose boundaries shift with numerical precision. Our findings emphasize that rigorous arithmetic precision is the key to dependable PDE solving with neural networks.
Paper Structure (24 sections, 16 equations, 8 figures, 2 tables)

This paper contains 24 sections, 16 equations, 8 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Related Works
  3. Problem and Experiment Settings
  4. Problem Settings
  5. Experiments Settings
  6. Understanding PINN Failure Modes from Loss Landscape Perspective
  7. Loss-Barrier Hypothesis vs Same-Basin Hypothesis: Which is the True Story?
  8. Physics-Informed Neural Networks Follow Same-Basin Hypothesis
  9. The Loss-Error Dynamics of PINN's Training
  10. Precision Serves a Key Factor in PINN's Training
  11. Failure Modes Disappear when Using Double-Precision Training
  12. Optimization Stops Early without Sufficient Arithmetic Precision
  13. Harder Equations Require Higher Arithmetic Precision
  14. Arithmetic Precision's Impact on Various Model Architectures.
  15. Discussion of Computation
  16. ...and 9 more sections

Figures (8)

  • Figure 1: Training Dynamic of PINN's failure mode case and success case.
  • Figure 2: Some cases of PINN failure modes.
  • Figure 3: A schematic that illustrates two kinds of loss landscape hypothesis.
  • Figure 4: Several empirical results on convection support Same-Basin Hypothesis.
  • Figure 5: Loss-Error Dynamics of PINN models can be in three phases.
  • ...and 3 more figures