Physics-informed Neural Networks for Functional Differential Equations: Cylindrical Approximation and Its Convergence Guarantees

TL;DR

This work provides a strong backbone for physicists, mathematicians, and machine learning experts to analyze previously challenging FDEs, thereby democratizing their numerical analysis, which has received limited attention.

Abstract

We propose the first learning scheme for functional differential equations (FDEs). FDEs play a fundamental role in physics, mathematics, and optimal control. However, the numerical analysis of FDEs has faced challenges due to its unrealistic computational costs and has been a long standing problem over decades. Thus, numerical approximations of FDEs have been developed, but they often oversimplify the solutions. To tackle these two issues, we propose a hybrid approach combining physics-informed neural networks (PINNs) with the \textit{cylindrical approximation}. The cylindrical approximation expands functions and functional derivatives with an orthonormal basis and transforms FDEs into high-dimensional PDEs. To validate the reliability of the cylindrical approximation for FDE applications, we prove the convergence theorems of approximated functional derivatives and solutions. Then, the derived high-dimensional PDEs are numerically solved with PINNs. Through the capabilities of PINNs, our approach can handle a broader class of functional derivatives more efficiently than conventional discretization-based methods, improving the scalability of the cylindrical approximation. As a proof of concept, we conduct experiments on two FDEs and demonstrate that our model can successfully achieve typical $L^1$ relative error orders of PINNs $\sim 10^{-3}$. Overall, our work provides a strong backbone for physicists, mathematicians, and machine learning experts to analyze previously challenging FDEs, thereby democratizing their numerical analysis, which has received limited attention. Code is available at \url{https://github.com/TaikiMiyagawa/FunctionalPINN}.

Figures (29)

• Figure 1: Overall architecture. An FDE is simplified to a high-dimensional PDE via the cylindrical approximation. The PDE is solved with a PINN. The approximated functional derivative can be efficiently computed with automatic differentiation.
• Figure 2: Cylindrical approximation of FDE's solution. The $L^1$ relative error, defined as $\frac{1}{N} \sum_{i=1}^{N} | F([\theta_i]) - F([P_m \theta_i]) | / | F([\theta_i]) |$, diminishes with increasing $m$. The Burgers-Hopf equation with the delta initial condition (Sec. \ref{['sec: Burgers-Hopf Equation']}) is considered. Note that PINN's training is not included.
• Figure 3: PINN's architecture.
• Figure 4: Analytic solution (top four panels), prediction (second four panels), and absolute error (bottom four panels) of FTE with degree 100 under linear initial condition. The horizontal axes represent $a_k$ for $k=0, 1, 2, 99$, with all the other coefficients set to 0. Our model successfully learns the FTE.
• Figure 5: Absolute error of first-order functional derivative of FTE with degree 100 (top) and 1000 (bottom) under linear (top) and nonlinear (bottom) initial conditions. The error bars represent the standard deviation over 10 runs with different random seeds.

Theorems & Definitions (37)

• Theorem 3.1: Pointwise convergence of approximated functional derivatives (informal)
• Theorem 3.2: Convergence of approximated solutions (informal)
• Definition C.1: Pointwise continuity of $F$
• Definition C.2: Uniform continuity of $F$
• Definition C.3: Compactness of $F$
• Definition C.4: Complete continuity of $F$
• Definition C.5: Boundedness of metric space of functions
• Definition C.6: Closedness of metric space of functions
• Definition C.7: Compactness of metric space of functions
• Definition C.8: Pre-compactness of metric space of functions