Chebyshev Feature Neural Network for Accurate Function Approximation

TL;DR

The paper tackles the challenge that standard deep neural networks often fail to reach machine-precision accuracy for high-precision scientific computing tasks. It introduces the Chebyshev Feature Neural Network (CFNN), which uses Chebyshev features with learnable frequencies in the first layer and a multi-stage residual training scheme, aided by stage-specific exponential initialization and a boundary-aware loss. The method achieves machine-precision training for a broad class of functions, including smooth and several nonsmooth cases, up to dimensions as high as $d=20$, illustrating strong accuracy and scalability. This work offers a practical, high-precision alternative to traditional DNNs, with potential impact on physics-informed learning and long-term predictive accuracy in scientific applications.

Abstract

We present a new Deep Neural Network (DNN) architecture capable of approximating functions up to machine accuracy. Termed Chebyshev Feature Neural Network (CFNN), the new structure employs Chebyshev functions with learnable frequencies as the first hidden layer, followed by the standard fully connected hidden layers. The learnable frequencies of the Chebyshev layer are initialized with exponential distributions to cover a wide range of frequencies. Combined with a multi-stage training strategy, we demonstrate that this CFNN structure can achieve machine accuracy during training. A comprehensive set of numerical examples for dimensions up to $20$ are provided to demonstrate the effectiveness and scalability of the method.

Figures (9)

• Figure 1: Architechture of the proposed CFNN.
• Figure 1: Results of $f_1(x)$. Left top: training loss history across all stages; Left bottom: test data ground truth vs prediction and prediction error; Right: training results at four stages showing the network output (red dashed) against the ground truth/residual (blue solid) at each stage.
• Figure 2: Results of $f_2(x)$. Left top: training loss history across all stages; Left bottom: test data ground truth vs prediction and prediction error; Right: training results at four stages showing the network output (red dashed) against the ground truth/residual (blue solid) at each stage.
• Figure 3: Results of $f_3(x)$. Left top: training loss history across all stages; Left bottom: test data ground truth vs prediction and prediction error; Right: training results at four stages showing the network output (red dashed) against the ground truth/residual (blue solid) at each stage.
• Figure 4: Results of $f_4(x)$. Left top: training loss history across all stages; Left bottom: test data ground truth vs prediction and prediction error; Right: training results at four stages showing the network output (red dashed) against the ground truth/residual (blue solid) at each stage.