Data vs. Physics: The Apparent Pareto Front of Physics-Informed Neural Networks

TL;DR

This work addresses why physics-informed neural networks (PINNs) often require careful loss weighting by showing that system parameters such as characteristic length and time scales, domain size, and PDE coefficients scale data and physics residuals differently. Using a two-system PINN framework applied to diffusion and Navier–Stokes equations under dynamic similitude, the authors derive that the residual scale ratio satisfies $f_i/e_i \sim \kappa/L^2$ for diffusion and $f_i/e_i \sim 1/L^2$ (and $f_i/e_i \sim 1/L^2$ for NS in nondimensional form), which directly biases multi-objective optimization. They define and empirically map the apparent Pareto front as the set of loss values achievable by gradient-based training across varying loss weights $\alpha$, showing that appropriate weighting can compensate parameter-induced imbalances and recover physically valid solutions, with parameterization sometimes introducing locally convex regions that widen the viable weight range. The findings provide practical guidance for loss weighting schemes and emphasize that nondimensionalization and parameter choices can significantly influence PINN training dynamics and outcomes, informing more robust MO optimization in physics-driven learning.

Abstract

Physics-informed neural networks (PINNs) have emerged as a promising deep learning method, capable of solving forward and inverse problems governed by differential equations. Despite their recent advance, it is widely acknowledged that PINNs are difficult to train and often require a careful tuning of loss weights when data and physics loss functions are combined by scalarization of a multi-objective (MO) problem. In this paper, we aim to understand how parameters of the physical system, such as characteristic length and time scales, the computational domain, and coefficients of differential equations affect MO optimization and the optimal choice of loss weights. Through a theoretical examination of where these system parameters appear in PINN training, we find that they effectively and individually scale the loss residuals, causing imbalances in MO optimization with certain choices of system parameters. The immediate effects of this are reflected in the apparent Pareto front, which we define as the set of loss values achievable with gradient-based training and visualize accordingly. We empirically verify that loss weights can be used successfully to compensate for the scaling of system parameters, and enable the selection of an optimal solution on the apparent Pareto front that aligns well with the physically valid solution. We further demonstrate that by altering the system parameterization, the apparent Pareto front can shift and exhibit locally convex parts, resulting in a wider range of loss weights for which gradient-based training becomes successful. This work explains the effects of system parameters on MO optimization in PINNs, and highlights the utility of proposed loss weighting schemes.

Figures (4)

• Figure 1: Geometrical setup and reference solution for the diffusion example. Representative sample of training data is shown as black crosses (IC and BC) and white circles (collocation).
• Figure 2: Geometrical setup and reference solution for the Navier-Stokes example. Representative sample of training data is shown as black crosses (IC and BC) and white circles (collocation).
• Figure 3: Data vs. physics loss (top row) and test set errors (bottom row) for the diffusion equation. Different system parameterizations are arranged as columns. The system parameters determine the residual scale ratio $f_i/e_i\sim\kappa/L^2$, where for $\kappa/L^2 \gg 1$ the scale of physics residuals is predominant and for $\kappa/L^2 \ll 1$ that of data residuals. Final optimization steps (empty circles) in the top row outline the apparent Pareto front.
• Figure 4: Data vs. physics loss (top row) and test set errors (bottom row) for the Navier-Stokes equations. Different system parameterizations are arranged as columns. The system parameters determine the residual scale ratio $f_i/e_i\sim1/L^2$, where for $1/L^2 \gg 1$ the scale of physics residuals is predominant and for $1/L^2 \ll 1$ that of data residuals. Final optimization steps (empty circles) in the top row outline the apparent Pareto front.