Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

About rectified sigmoid function for enhancing the accuracy of Physics-Informed Neural Networks

Vasiliy A. Es'kin, Alexey O. Malkhanov, Mikhail E. Smorkalov

TL;DR

This work tackles the accuracy limitations of physics-informed neural networks (PINNs) when solving ODEs with shallow architectures. It introduces a rectified sigmoid activation $Re-σ(x)$ and adapts physics-informed data-driven (PIDD) initialization as well as neuron-by-neuron (NbN) gradient-free training to a one-hidden-layer PINN framework. Across three benchmark problems—the harmonic oscillator, relativistic slingshot, and Lorenz system—the rectified activation yields substantial relative $L_2$ error reductions compared with the traditional sigmoid activation, with consistent benefits for both initialization and training phases. The results demonstrate that shallow PINNs with tailored activations can achieve accuracy close to standard numerical solvers on diverse dynamical systems, suggesting practical benefits for efficient physics-informed inference and potential extensions to more complex models.

Abstract

The article is devoted to the study of neural networks with one hidden layer and a modified activation function for solving physical problems. A rectified sigmoid activation function has been proposed to solve physical problems described by the ODE with neural networks. Algorithms for physics-informed data-driven initialization of a neural network and a neuron-by-neuron gradient-free fitting method have been presented for the neural network with this activation function. Numerical experiments demonstrate the superiority of neural networks with a rectified sigmoid function over neural networks with a sigmoid function in the accuracy of solving physical problems (harmonic oscillator, relativistic slingshot, and Lorentz system).

About rectified sigmoid function for enhancing the accuracy of Physics-Informed Neural Networks

TL;DR

This work tackles the accuracy limitations of physics-informed neural networks (PINNs) when solving ODEs with shallow architectures. It introduces a rectified sigmoid activation and adapts physics-informed data-driven (PIDD) initialization as well as neuron-by-neuron (NbN) gradient-free training to a one-hidden-layer PINN framework. Across three benchmark problems—the harmonic oscillator, relativistic slingshot, and Lorenz system—the rectified activation yields substantial relative error reductions compared with the traditional sigmoid activation, with consistent benefits for both initialization and training phases. The results demonstrate that shallow PINNs with tailored activations can achieve accuracy close to standard numerical solvers on diverse dynamical systems, suggesting practical benefits for efficient physics-informed inference and potential extensions to more complex models.

Abstract

The article is devoted to the study of neural networks with one hidden layer and a modified activation function for solving physical problems. A rectified sigmoid activation function has been proposed to solve physical problems described by the ODE with neural networks. Algorithms for physics-informed data-driven initialization of a neural network and a neuron-by-neuron gradient-free fitting method have been presented for the neural network with this activation function. Numerical experiments demonstrate the superiority of neural networks with a rectified sigmoid function over neural networks with a sigmoid function in the accuracy of solving physical problems (harmonic oscillator, relativistic slingshot, and Lorentz system).
Paper Structure (13 sections, 13 equations, 1 figure, 2 tables, 4 algorithms)

This paper contains 13 sections, 13 equations, 1 figure, 2 tables, 4 algorithms.

Table of Contents

  1. Introduction
  2. State of the problem
  3. Numerical experiments
  4. Set of physical problems
  5. Harmonic Oscillator
  6. Relativistic slingshot
  7. Lorentz system
  8. Results of numerical experiments for the PIDD initialization
  9. Results of numerical experiments for the NbN training
  10. Conclusions
  11. Physics-informed data-driven (PIDD) initialization of neural networks
  12. Neuron-by-neuron training of neural networks
  13. Measure of accuracy

Figures (1)

  • Figure 1: (a), (b), (c) and (d) are comparison of the predicted (red dash lines) and reference solutions (blue solid lines) corresponding to $h$, $x$, $y$ and $z$, respectively. (e), (f), (g) and (h) are absolute errors $|h^{\text{ref}} - h_{\bm \theta}|$, $|x^{\text{ref}} - x_{\bm \theta}|$, $|y^{\text{ref}} - y_{\bm \theta}|$, and $|z^{\text{ref}} - z_{\bm \theta}|$, respectively.