Higher-order-ReLU-KANs (HRKANs) for solving physics-informed neural networks (PINNs) more accurately, robustly and faster

TL;DR

Detailed experiments on two famous and representative PDEs, namely, the linear Poisson equation and nonlinear Burgers’ equation with viscosity, reveal that the proposed Higher-order-ReLU-KANs (HRKANs) achieve the highest fitting accuracy and training robustness and lowest training time significantly among KANs, ReLU-KANs and HRKANs.

Abstract

Finding solutions to partial differential equations (PDEs) is an important and essential component in many scientific and engineering discoveries. One of the common approaches empowered by deep learning is Physics-informed Neural Networks (PINNs). Recently, a new type of fundamental neural network model, Kolmogorov-Arnold Networks (KANs), has been proposed as a substitute of Multilayer Perceptions (MLPs), and possesses trainable activation functions. To enhance KANs in fitting accuracy, a modification of KANs, so called ReLU-KANs, using "square of ReLU" as the basis of its activation functions, has been suggested. In this work, we propose another basis of activation functions, namely, Higherorder-ReLU (HR), which is simpler than the basis of activation functions used in KANs, namely, Bsplines; allows efficient KAN matrix operations; and possesses smooth and non-zero higher-order derivatives, essential to physicsinformed neural networks. We name such KANs with Higher-order-ReLU (HR) as their activations, HRKANs. Our detailed experiments on two famous and representative PDEs, namely, the linear Poisson equation and nonlinear Burgers' equation with viscosity, reveal that our proposed Higher-order-ReLU-KANs (HRKANs) achieve the highest fitting accuracy and training robustness and lowest training time significantly among KANs, ReLU-KANs and HRKANs. The codes to replicate our experiments are available at https://github.com/kelvinhkcs/HRKAN.

Figures (7)

• Figure 1: B-Spline basis $B_{5,3}$ (left) and basis function $b_4(x)$ (right)
• Figure 2: "Square of ReLU" basis $R_{5,3}$ (left) and basis function $r_4(x)$ (right)
• Figure 3: The "square of ReLU" $r_4(x)$ and its first and second-order derivatives $r'_4(x)$ and $r"_4(x)$ (left) and the Higher-order-ReLU $v_{4,4}(x)$ and its first and second-order derivatives $v'_{4,4}(x)$ and $v"_{4,4}(x)$ (right).
• Figure 4: The ground-truth solution and the solutions learnt by the KAN, ReLU-KAN and HRKAN (Top row) and their residual difference compared against the ground-truth solution (Bottom row) in one of the 10 runs with the Poisson equation.
• Figure 5: The median and max-min-band of training loss, test loss and test MSE (MSE between the test-set ground-truth solution and the learnt solutions) of 10 runs for the KAN, ReLU-KAN and HRKAN with the Poisson equation. First column: all 3000 epoches; Second column: last 2000 epoches; Last column: last 1000 epoches.