Learning and discovering multiple solutions using physics-informed neural networks with random initialization and deep ensemble

TL;DR

The paper addresses the challenge of solution multiplicity in nonlinear differential equations by showing that physics-informed neural networks (PINNs) with random initialization and deep ensembles can uncover multiple solution patterns. A hybrid workflow clusters ensemble PINN outputs to obtain representative solutions that serve as initial conditions or guesses for high-fidelity solvers (FDM/FEM/SEM), enabling efficient refinement and verification of distinct modes, including unstable ones. The authors demonstrate this approach on 1D Bratu, boundary-layer, reaction-diffusion, and 2D Allen–Cahn and cavity-flow problems, showing that PINN ensembles can expose diverse solutions and guide classical solvers to robust, convergent results. This framework offers a general, practical method to explore nonlinear DE solution multiplicity and to leverage PINNs as a discovery tool that complements traditional numerical methods.

Abstract

We explore the capability of physics-informed neural networks (PINNs) to discover multiple solutions. Many real-world phenomena governed by nonlinear differential equations (DEs), such as fluid flow, exhibit multiple solutions under the same conditions, yet capturing this solution multiplicity remains a significant challenge. A key difficulty is giving appropriate initial conditions or initial guesses, to which the widely used time-marching schemes and Newton's iteration method are very sensitive in finding solutions for complex computational problems. While machine learning models, particularly PINNs, have shown promise in solving DEs, their ability to capture multiple solutions remains underexplored. In this work, we propose a simple and practical approach using PINNs to learn and discover multiple solutions. We first reveal that PINNs, when combined with random initialization and deep ensemble method -- originally developed for uncertainty quantification -- can effectively uncover multiple solutions to nonlinear ordinary and partial differential equations (ODEs/PDEs). Our approach highlights the critical role of initialization in shaping solution diversity, addressing an often-overlooked aspect of machine learning for scientific computing. Furthermore, we propose utilizing PINN-generated solutions as initial conditions or initial guesses for conventional numerical solvers to enhance accuracy and efficiency in capturing multiple solutions. Extensive numerical experiments, including the Allen-Cahn equation and cavity flow, where our approach successfully identifies both stable and unstable solutions, validate the effectiveness of our method. These findings establish a general and efficient framework for addressing solution multiplicity in nonlinear differential equations.

Figures (17)

• Figure 1: A schematic illustration of the proposed approach for solving ODEs/PDEs with solution multiplicity. (a) Traditional applications of PINNs typically focus on obtaining a single solution by optimizing the PINN loss function $\mathcal{L}(\theta)$ with respect to the NN parameters $\theta$. (b) When applied to ODEs/PDEs with multiple solutions, the approach consists of three key steps: (1) PINNs are randomly initialized to generate an ensemble of $M$ PINN solutions $\{u_{\theta^*_i}\}_{i=1}^M$; (2) these solutions are clustered based on their distinct patterns or modes and downsampled to a set of representative PINN solutions, $\{u_{\theta_i^*}\}_{i\in I}$ where $I$ denotes their index set; and (3) these representative solutions serve as initial conditions or guesses for conventional numerical solvers (FDM-, FEM-, or SEM-based), refining them into highly accurate solutions, $\{u_i^*\}_{i\in J}$, with high precision and guaranteed convergence.
• Figure 2: Solving a 1D steady-state PDE using PINNs with $1,000$ randomly initialized NNs, which results in the same approximation of the source term $f$ (on the left) but two distinct approximations of the solution $u$ (in the middle). These two distinct PINN solutions are then numerically verified by using them as the initial guesses for a finite-difference-method-based solver on the uniform mesh with mesh size $h=1/1600$ (on the right). Details and more results can be found in Section 3.3.
• Figure 3: Solving the 1D Bratu problem using PINNs with (a) ten randomly initialized NNs and (b) different values of $\lambda$ (from right to left are $\lambda=3.5, 2, 1, 0.5$). For each value of $\lambda$, ten PINNs are trained starting from these ten randomly initialized NNs displayed in (a).
• Figure 4: Training of PINNs in solving the 1D Bratu problem with $\lambda=1$ and $1,000$ randomly initialized NNs. In (a), we show the bifurcation of the predicted values of $u(0.5)$ across the $1,000$ NNs. In (b), we present PINN solutions at different iterations. In (c) and (d), we display the PINN loss for these $1,000$ NNs.
• Figure 5: Results for solving the 1D Bratu problem with $\lambda=1$ of (a) using representative PINN solutions at 100th iteration as the initial guesses for a BVP numerical solver and (b) parameter sharing PINNs.