Constrained or Unconstrained? Neural-Network-Based Equation Discovery from Data

TL;DR

The paper tackles the problem of discovering governing PDEs from noisy data by representing the unknown PDE as a neural network $\mathcal{N}^{\phi}$ and using a denoised state $u^{\theta}$ as an intermediary. It compares two gradient-based training paradigms—a penalty method and a constrained, barrier-based approach—solving the learned PDE with classical numerical methods via the method of lines. The constrained approach, particularly with a trust-region barrier formulation, shows improved robustness to noise and fewer collocation points across Burgers' and KdV equations, albeit at higher computational cost. The work demonstrates that PDE discovery can be effectively integrated with classical solvers and mesh-robust validation, offering a viable alternative to PINNs in data-driven equation discovery with practical impact for modeling nonlinear dynamics.

Abstract

Throughout many fields, practitioners often rely on differential equations to model systems. Yet, for many applications, the theoretical derivation of such equations and/or accurate resolution of their solutions may be intractable. Instead, recently developed methods, including those based on parameter estimation, operator subset selection, and neural networks, allow for the data-driven discovery of both ordinary and partial differential equations (PDEs), on a spectrum of interpretability. The success of these strategies is often contingent upon the correct identification of representative equations from noisy observations of state variables and, as importantly and intertwined with that, the mathematical strategies utilized to enforce those equations. Specifically, the latter has been commonly addressed via unconstrained optimization strategies. Representing the PDE as a neural network, we propose to discover the PDE by solving a constrained optimization problem and using an intermediate state representation similar to a Physics-Informed Neural Network (PINN). The objective function of this constrained optimization problem promotes matching the data, while the constraints require that the PDE is satisfied at several spatial collocation points. We present a penalty method and a widely used trust-region barrier method to solve this constrained optimization problem, and we compare these methods on numerical examples. Our results on the Burgers' and the Korteweg-De Vreis equations demonstrate that the latter constrained method outperforms the penalty method, particularly for higher noise levels or fewer collocation points. For both methods, we solve these discovered neural network PDEs with classical methods, such as finite difference methods, as opposed to PINNs-type methods relying on automatic differentiation. We briefly highlight other small, yet crucial, implementation details.

Figures (18)

• Figure 1: Using $N_r = 1000$ collocation points, the proposed constrained formulation outperforms the penalty-method approach, for nonzero noise levels. On a very similar problem, raissi_deep_2018 reported an error of $0.46$ for $\text{noise level} = 0.05$ for $u_{0}^{\mathrm{train}}$ (see Table 1). Recall that $u_{0}^{\mathrm{test}}$ is entirely unseen during training. Each box-and-whisker shows ten datasets / initializations and runs through Alg. \ref{['alg:train_validation']} with validation. In the 0.0 noise level case, the location of the data, the location of the collocation points, and the neural network initializations change (and the initialization of $\lambda_j$ for the penalty-method approach).
• Figure 2: With the same setup and runs as Fig. \ref{['fig:burgers:noise_level:l2']}, we show the time-to-failure (Eq. (\ref{['eq:results:ttf']})). Higher is better for this performance measure. Note that for $u_{0}^{\mathrm{train}}$, $T=30$, but for $u_{0}^{\mathrm{test}}$, $T = 10$.
• Figure 3: Despite noisy training data and relatively few collocation points, the constrained method accurately recovers the governing PDE. We show the solution of PDE discovered by the constrained method, for the highest noise level ($0.4$), with the lowest training error in Fig. \ref{['fig:burgers:noise_level:l2']}. This is the PDE solution for the $u_{0}^{\mathrm{train}}$ initial condition. The $u_{0}^{\mathrm{test}}$ initial condition for the same discovered PDE is shown in Fig. \ref{['fig:burgers:example:con:extrap']}.
• Figure 4: For the constrained method, the discovered PDE accurately predicts the evolution of an unseen initial condition. The PDE discovered in Fig. \ref{['fig:burgers:example:con:train']} is solved again with a classical finite difference method, but for $u_{0}^{\mathrm{test}}$, while all of the (noisy) training data came originally from $u_{0}^{\mathrm{train}}$. In other words, the noisy data from $u_{0}^{\mathrm{train}}$ (middle plot of Fig. \ref{['fig:burgers:example:con:train']}) is used to train $\mathcal{N}^{\phi}$, and the initial and boundary conditions for the new $u_{0}^{\mathrm{test}}$ (middle plot above), are used for the numerical solution in the right plot above. This run corresponds to $\ell^{2}_{\mathrm{rel}} = 0.06$ for $u_{0}^{\mathrm{test}}$ in Fig. \ref{['fig:burgers:noise_level:l2']}.
• Figure 5: Similar to Fig. \ref{['fig:burgers:example:con:train']}, we show the solution of PDE discovered through the penalty-method approach, for the highest noise level ($0.4$), with the lowest training error in Fig. \ref{['fig:burgers:noise_level:l2']}. This is the PDE solution for the $u_{0}^{\mathrm{train}}$ initial condition. The $u_{0}^{\mathrm{test}}$ initial condition for the same discovered PDE is shown in Fig. \ref{['fig:burgers:example:sa:extrap']}. Here, we see some asymmetry, with the $u(x,t) = 0$ contour slightly drifting from the $x=0$ axis.