Stacked networks improve physics-informed training: applications to neural networks and deep operator networks

TL;DR

This work addresses the difficulty of training physics-informed neural networks and physics-informed DeepONets on challenging dynamical systems by introducing a stacking multifidelity framework. The method builds a chain of networks where each level uses the previous output as a low-fidelity input, and can employ the same or a progressively modified equation, effectively implementing curriculum-like learning. Demonstrations on a damped pendulum, multiscale forcing, the wave equation, and Burgers-type dynamics show that stacking improves accuracy, enables solutions with smaller networks, and enhances robustness where single-fidelity approaches fail. The approach is compatible with neural tangent kernel weighting and can be extended to various architectures, offering a flexible pathway to more reliable physics-informed learning with limited data.

Abstract

Physics-informed neural networks and operator networks have shown promise for effectively solving equations modeling physical systems. However, these networks can be difficult or impossible to train accurately for some systems of equations. We present a novel multifidelity framework for stacking physics-informed neural networks and operator networks that facilitates training. We successively build a chain of networks, where the output at one step can act as a low-fidelity input for training the next step, gradually increasing the expressivity of the learned model. The equations imposed at each step of the iterative process can be the same or different (akin to simulated annealing). The iterative (stacking) nature of the proposed method allows us to progressively learn features of a solution that are hard to learn directly. Through benchmark problems including a nonlinear pendulum, the wave equation, and the viscous Burgers equation, we show how stacking can be used to improve the accuracy and reduce the required size of physics-informed neural networks and operator networks.

Figures (19)

• Figure 1: Graphical abstract. A previous prediction is used as the low-fidelity input for a physics-informed multifidelity neural network to generate a more accurate prediction as the output.
• Figure 2: Physics-informed neural network
• Figure 3: PINN results $s_1$ (left) and $s_2$ (right) as a function of time for the pendulum problem for ten random initial seedings. In each case, the PINN solution decays and does not agree well with the exact solution.
• Figure 4: Multifidelity physics-informed neural network
• Figure 5: Stacking multifidelity physics-informed neural network