Algorithmically Designed Artificial Neural Networks (ADANNs): Higher order deep operator learning for parametric partial differential equations

TL;DR

The paper addresses the challenge of learning operators for parametric PDEs by proposing Algorithmically Designed Artificial Neural Networks (ADANNs), which fuse base models that mimic efficient numerical algorithms with trainable difference models to correct residuals. The three core components—specialized base initializations, a learnable difference module, and optimization over base initializations—enable higher-order operator learning that can overcome conventional limits of deep operator methods. Demonstrations across semilinear heat, Sine-Gordon, viscous Burgers, and reaction-diffusion equations show that ADANNs can outperform classical numerical solvers and existing operator-learning approaches, using a two-tier training regime and black-box optimization to select effective initializations. The work highlights the potential for principled integration of numerical analysis and deep learning to improve parametric PDE solvers, with avenues for broader PDE classes and theoretical analysis left for future work.

Abstract

In this article we propose a new deep learning approach to approximate operators related to parametric partial differential equations (PDEs). In particular, we introduce a new strategy to design specific artificial neural network (ANN) architectures in conjunction with specific ANN initialization schemes which are tailor-made for the particular approximation problem under consideration. In the proposed approach we combine efficient classical numerical approximation techniques with deep operator learning methodologies. Specifically, we introduce customized adaptions of existing ANN architectures together with specialized initializations for these ANN architectures so that at initialization we have that the ANNs closely mimic a chosen efficient classical numerical algorithm for the considered approximation problem. The obtained ANN architectures and their initialization schemes are thus strongly inspired by numerical algorithms as well as by popular deep learning methodologies from the literature and in that sense we refer to the introduced ANNs in conjunction with their tailor-made initialization schemes as Algorithmically Designed Artificial Neural Networks (ADANNs). We numerically test the proposed ADANN methodology in the case of several parametric PDEs. In the tested numerical examples the ADANN methodology significantly outperforms existing traditional approximation algorithms as well as existing deep operator learning methodologies from the literature.

Figures (16)

• Figure 1: Graphical illustration for the base model defined in \ref{['semilinear_heat:eq11']} and \ref{['semilinear_heat:eq12']}.
• Figure 2: Graphical illustration of the performance of the methods in \ref{['table:SineGordon1d']}.
• Figure 3: Illustration of the methodology without difference model (cf. \ref{['adanns_pseudocode_nodiff']}) with a grid-based black box optimizer applied to the approximation of the operator in \ref{['SG:eq2']} based on the Sine-Gordon-type equation in \ref{['SG:eq1']} in the case $d=1$. Left: Test errors of the base models prior to training as a function of the parameters used for initialization. Right: Test errors of the trained base models as a function of the parameters used for initialization.
• Figure 4: Illustration of the methodology without difference model (cf. \ref{['adanns_pseudocode_nodiff']}) with our heuristic exploration-exploitation black box optimizer applied to the approximation of the operator in \ref{['SG:eq2']} mapping initial values to terminal values of the Sine-Gordon-type equation in \ref{['SG:eq1']} in the case $d=1$. Left: Test errors of trained base models as a function of the parameters used for initialization. Increasing scatter sizes indicate higher training run numbers. Middle: The same test errors represented in the order in which they appeared in the black box optimization process. Right: Coordinates of the chosen parameters in the black box optimization process.
• Figure 5: Example approximation plots for a randomly chosen initial value for the Sine-Gordon-type equation in \ref{['SG:eq1']} in the case $d=1$. Left: and approximations. Right: Classical and approximations.