Challenges in Training PINNs: A Loss Landscape Perspective
Pratik Rathore, Weimu Lei, Zachary Frangella, Lu Lu, Madeleine Udell
TL;DR
The paper analyzes why PINN training is difficult by linking ill-conditioning of differential operators to the loss landscape, formalizing the population Gauss-Newton matrix and its concentration to the empirical GN matrix under ridge incoherence. It develops optimization-theoretic results showing that, with standard regularity and Polyak-Lojasiewicz (PL*) conditions, gradient descent makes linear progress to a neighborhood of a minimizer and damped Newton achieves fast local convergence within a neighborhood. The authors relate the conditioning to two eigen-decay scenarios, illustrating when optimization becomes hard and how coupling first- and second-order methods improves convergence. They discuss practical optimization strategies and the potential benefits of combining Adam with L-BFGS or using second-order solvers such as NysNewton-CG to enhance PINN performance on difficult PDEs.
Abstract
This paper explores challenges in training Physics-Informed Neural Networks (PINNs), emphasizing the role of the loss landscape in the training process. We examine difficulties in minimizing the PINN loss function, particularly due to ill-conditioning caused by differential operators in the residual term. We compare gradient-based optimizers Adam, L-BFGS, and their combination Adam+L-BFGS, showing the superiority of Adam+L-BFGS, and introduce a novel second-order optimizer, NysNewton-CG (NNCG), which significantly improves PINN performance. Theoretically, our work elucidates the connection between ill-conditioned differential operators and ill-conditioning in the PINN loss and shows the benefits of combining first- and second-order optimization methods. Our work presents valuable insights and more powerful optimization strategies for training PINNs, which could improve the utility of PINNs for solving difficult partial differential equations.