Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

ConFIG: Towards Conflict-free Training of Physics Informed Neural Networks

Qiang Liu, Mengyu Chu, Nils Thuerey

TL;DR

The paper addresses gradient conflicts in Physics-Informed Neural Networks (PINNs) arising from multiple loss terms, such as PDE residuals and boundary/initial conditions. It introduces ConFIG, which computes a conflict-free update by ensuring positive projections onto all loss gradients via a unit-normalized gradient matrix and its pseudoinverse, with an adaptive length rule; a momentum-accelerated variant (M-ConFIG) further reduces backpropagation cost by alternating gradient updates. The authors prove convergence under convex and certain non-convex settings and demonstrate superior performance and runtime efficiency of ConFIG and M-ConFIG across challenging PINN tasks (Burgers, Schrödinger, Kovasznay, Beltrami), high-dimensional PDEs, and multi-task learning benchmarks (CelebA). The methods yield more balanced optimization across loss terms, mitigate bias toward the PDE residual, and offer practical impact for robust PINN training and scalable multi-task optimization.

Abstract

The loss functions of many learning problems contain multiple additive terms that can disagree and yield conflicting update directions. For Physics-Informed Neural Networks (PINNs), loss terms on initial/boundary conditions and physics equations are particularly interesting as they are well-established as highly difficult tasks. To improve learning the challenging multi-objective task posed by PINNs, we propose the ConFIG method, which provides conflict-free updates by ensuring a positive dot product between the final update and each loss-specific gradient. It also maintains consistent optimization rates for all loss terms and dynamically adjusts gradient magnitudes based on conflict levels. We additionally leverage momentum to accelerate optimizations by alternating the back-propagation of different loss terms. We provide a mathematical proof showing the convergence of the ConFIG method, and it is evaluated across a range of challenging PINN scenarios. ConFIG consistently shows superior performance and runtime compared to baseline methods. We also test the proposed method in a classic multi-task benchmark, where the ConFIG method likewise exhibits a highly promising performance. Source code is available at https://tum-pbs.github.io/ConFIG

ConFIG: Towards Conflict-free Training of Physics Informed Neural Networks

TL;DR

The paper addresses gradient conflicts in Physics-Informed Neural Networks (PINNs) arising from multiple loss terms, such as PDE residuals and boundary/initial conditions. It introduces ConFIG, which computes a conflict-free update by ensuring positive projections onto all loss gradients via a unit-normalized gradient matrix and its pseudoinverse, with an adaptive length rule; a momentum-accelerated variant (M-ConFIG) further reduces backpropagation cost by alternating gradient updates. The authors prove convergence under convex and certain non-convex settings and demonstrate superior performance and runtime efficiency of ConFIG and M-ConFIG across challenging PINN tasks (Burgers, Schrödinger, Kovasznay, Beltrami), high-dimensional PDEs, and multi-task learning benchmarks (CelebA). The methods yield more balanced optimization across loss terms, mitigate bias toward the PDE residual, and offer practical impact for robust PINN training and scalable multi-task optimization.

Abstract

The loss functions of many learning problems contain multiple additive terms that can disagree and yield conflicting update directions. For Physics-Informed Neural Networks (PINNs), loss terms on initial/boundary conditions and physics equations are particularly interesting as they are well-established as highly difficult tasks. To improve learning the challenging multi-objective task posed by PINNs, we propose the ConFIG method, which provides conflict-free updates by ensuring a positive dot product between the final update and each loss-specific gradient. It also maintains consistent optimization rates for all loss terms and dynamically adjusts gradient magnitudes based on conflict levels. We additionally leverage momentum to accelerate optimizations by alternating the back-propagation of different loss terms. We provide a mathematical proof showing the convergence of the ConFIG method, and it is evaluated across a range of challenging PINN scenarios. ConFIG consistently shows superior performance and runtime compared to baseline methods. We also test the proposed method in a classic multi-task benchmark, where the ConFIG method likewise exhibits a highly promising performance. Source code is available at https://tum-pbs.github.io/ConFIG
Paper Structure (45 sections, 2 theorems, 46 equations, 26 figures, 27 tables, 4 algorithms)

This paper contains 45 sections, 2 theorems, 46 equations, 26 figures, 27 tables, 4 algorithms.

Table of Contents

  1. Introduction
  2. Related Work
  3. Weighting Strategies for PINNs' Training
  4. Gradient Improvement Strategies in MTL and CL
  5. Method
  6. Conflict-free Inverse Gradients Method
  7. Two Term Losses and Positioning w.r.t. Existing Approaches
  8. ConFIG with Momentum Acceleration
  9. Experiments
  10. Physics Informed Neural Networks
  11. Preliminaries of PINNs.
  12. Focusing on two loss terms.
  13. Scenarios with three loss terms.
  14. Adjusting direction weights.
  15. Evaluation of Runtime Performance
  16. ...and 30 more sections

Key Result

Theorem 1

Assume that (a) $m$ objectives $\mathcal{L}_1,\mathcal{L}_2\cdots, \mathcal{L}_m$ are convex and differentiable; (b)The gradient $\bm{g}= \nabla_\theta\mathcal{L}=\sum_{i=1}^m \nabla_\theta\mathcal{L}_i$ is Lipschitz continuous with constant $L > 0$. Then, update along ConFIG direction $\bm{g}_{\tex

Figures (26)

  • Figure 1: Visualization of toy example showing the conflict between different losses during optimization.
  • Figure 2: Sketch of PCGrad (a), IMTL-G (b), and our ConFIG (c) method with two loss terms. The PCGrad method directly sums two orthogonal components, and the IMTL-G method rescales the two vectors to the same magnitude. Our ConFIG method sums the unit vector of the orthogonal components and adjusts its magnitude with the projection length of each loss-specific gradient.
  • Figure 3: Examples of PINN predictions and squared error (SE) distributions on the test PDEs.
  • Figure 4: Relative improvements of PINNs trained with two loss terms using different methods.
  • Figure 5: Training losses of PINNs trained with Adam baselines and ConFIG using two loss terms.
  • ...and 21 more figures

Theorems & Definitions (4)

  • Theorem 1
  • proof
  • Theorem 2
  • proof