A Kolmogorov High Order Deep Neural Network for High Frequency Partial Differential Equations in High Dimensions

TL;DR

This paper tackles the curse of dimensionality in solving high-frequency PDEs by marrying the high-order one-dimensional approximation of HOrderDNN with the Kolmogorov Superposition Theorem. The resulting K-HOrderDNN approximates multivariate functions via univariate inner functions and a univariate outer function, reducing parameter growth from $(p+1)^d$ to $(p+1)d$ while preserving, and often enhancing, accuracy for high-dimensional, high-frequency problems. Theoretical results show CoD avoidance for a dense subset of continuous functions, with explicit rates depending on the inner/outer approximations, and numerical experiments demonstrate superior performance over PINN and HOrderDNN across Poisson and Helmholtz equations in dimensions up to $d=50$. The approach offers faster convergence, better frequency handling, and scalable parameter counts, suggesting strong practical impact for high-dimensional PDE solvers in scientific computing. Future work aims to extend to more complex PDEs and improve sampling strategies to further enhance performance and robustness.

Abstract

This paper proposes a Kolmogorov high order deep neural network (K-HOrderDNN) for solving high-dimensional partial differential equations (PDEs), which improves the high order deep neural networks (HOrderDNNs). HOrderDNNs have been demonstrated to outperform conventional DNNs for high frequency problems by introducing a nonlinear transformation layer consisting of $(p+1)^d$ basis functions. However, the number of basis functions grows exponentially with the dimension $d$, which results in the curse of dimensionality (CoD). Inspired by the Kolmogorov superposition theorem (KST), which expresses a multivariate function as superpositions of univariate functions and addition, K-HOrderDNN utilizes a HOrderDNN to efficiently approximate univariate inner functions instead of directly approximating the multivariate function, reducing the number of introduced basis functions to $d(p+1)$. We theoretically demonstrate that CoD is mitigated when target functions belong to a dense subset of continuous multivariate functions. Extensive numerical experiments show that: for high-dimensional problems ($d$=10, 20, 50) where HOrderDNNs($p>1$) are intractable, K-HOrderDNNs($p>1$) exhibit remarkable performance. Specifically, when $d=10$, K-HOrderDNN($p=7$) achieves an error of 4.40E-03, two orders of magnitude lower than that of HOrderDNN($p=1$) (see Table 10); for high frequency problems, K-HOrderDNNs($p>1$) can achieve higher accuracy with fewer parameters and faster convergence rates compared to HOrderDNNs (see Table 8).

Figures (17)

• Figure 1: Illustration of the architecture of HOrderDNN($p=2$) with $d=2$. The only difference to conventional DNNs is the nonlinear transformation layer followed by the input layer.
• Figure 2: Schematic illustration of K-HOrderDNN($p=2$) for $d=2$. Compared to HOrderDNN($p=2$) decipted in Fig. \ref{['fig: HOrderDNN']}, tensor product basis functions are avoided to prevent parameter explosion. Note that the subnetwork $g\,=\,\mathbf{G}_{NN}\circ\sigma\circ G_1$. The width and depth of the subnetworks $h_p$ and $g$ are provided in Table \ref{['table:table1']}, and specific values can be found in the detailed examples.
• Figure 3: The change in relative $L_2$ errors with respect to hw (the first row) and the number of trainable parameters (the second row) for PINN, HOrderDNNs, and K-HOrderDNNs on problem (\ref{['equation2']}) when $d$=2. Each column corresponds to a different setting of $hd$, specifically $hd$ = 1, 2, and 3.
• Figure 4: The change in relative $L_2$ errors with respect to $gw$ (the first row) and the number of trainable parameters (the second row) for K-HOrderDNN($p$) on the problem (\ref{['equation2']}) when $d$=2. Each column corresponds to a different setting of $gd$, specifically $gd$ = 2, 3, and 4.
• Figure 5: Convergence processes of HOrderDNNs (the first row), K-HOrderDNNs (the second row), and PINN under different order p on the problem (\ref{['equation2']}) when d=2. Here, $hd$=3, $hw$=45, $gd$=2, $gw$=90 for K-HOrderDNNs, and $L$=6, $W$=90 for HOrderDNNs and PINN.

Theorems & Definitions (13)

• Theorem 1: Kolmogorov Superposition Theorem
• Lemma 1: Jackson Theoremwang2019numerical
• Lemma 2: lai2022kolmogorov
• Theorem 2
• proof
• Theorem 3
• Theorem 4
• proof
• Lemma E1
• proof