Weights initialization of neural networks for function approximation

TL;DR

The paper introduces a reusable weights initialization framework for neural function approximation that leverages pretrained basis networks on a reference domain and a domain-mapping mechanism to extend to arbitrary domains. By progressively reusing parameters across basis functions and mapping inputs to a canonical interval, the approach improves training efficiency, generalization, and transferability across tasks. Empirical results in one- and two-dimensional settings demonstrate strong accuracy, robust extrapolation, and favorable trade-offs between depth, width, and activation choices, with GELU identified as a practical default. The work also provides a modular software package and discusses future directions, including higher-dimensional problems, physics-informed constraints, and PDE applications.

Abstract

Neural network-based function approximation plays a pivotal role in the advancement of scientific computing and machine learning. Yet, training such models faces several challenges: (i) each target function often requires training a new model from scratch; (ii) performance is highly sensitive to architectural and hyperparameter choices; and (iii) models frequently generalize poorly beyond the training domain. To overcome these challenges, we propose a reusable initialization framework based on basis function pretraining. In this approach, basis neural networks are first trained to approximate families of polynomials on a reference domain. Their learned parameters are then used to initialize networks for more complex target functions. To enhance adaptability across arbitrary domains, we further introduce a domain mapping mechanism that transforms inputs into the reference domain, thereby preserving structural correspondence with the pretrained models. Extensive numerical experiments in one- and two-dimensional settings demonstrate substantial improvements in training efficiency, generalization, and model transferability, highlighting the promise of initialization-based strategies for scalable and modular neural function approximation. The full code is made publicly available on Gitee.

Figures (32)

• Figure 1: Extrapolation on $[-15, 15]$ from model trained on $[-10, 10]$.
• Figure 2: Training loss curves (log scale) under different initialization strategies.
• Figure 3: Workflow of the reusable weights initialization framework for function approximation. the left panel shows the pretraining of neural networks to approximate canonical basis functions on a reference domain. The learned weights are then reused to initialize the hidden layers of a general approximator (right panel), enabling faster convergence, improved generalization, and structural flexibility.
• Figure 4: Modular construction of basis neural networks. Each module in orange box denotes a neural approximation to a specific basis function $\varphi_k$. These modules can be reused and composed for downstream tasks.
• Figure 5: Model inference of $y = x^3$ over $[-60,60]$ after inverse mapping.

Theorems & Definitions (1)

• Remark 1