Brain-Inspired Physics-Informed Neural Networks: Bare-Minimum Neural Architectures for PDE Solvers

TL;DR

The paper addresses the question of how far PINN architectures can be simplified while still accurately solving PDEs. It merges Physics-Informed Neural Networks with Brain-Inspired Modular Training (BIMT) to produce bare-minimum, sparse, and modular networks that evolve during training, revealing a spectral bias where high-frequency components require more neural units. It provides concrete implementation details, derives primitive modular building blocks, and demonstrates that modular PINNs can outperform equivalent-capacity dense networks on certain problems. The work suggests substantial potential for resource-efficient PDE solvers, while acknowledging current sparse-computation limitations and outlining directions for expanding modular primitives and architectures.

Abstract

Physics-Informed Neural Networks (PINNs) have emerged as a powerful tool for solving partial differential equations~(PDEs) in various scientific and engineering domains. However, traditional PINN architectures typically rely on large, fully connected multilayer perceptrons~(MLPs), lacking the sparsity and modularity inherent in many traditional numerical solvers. An unsolved and critical question for PINN is: What is the minimum PINN complexity regarding nodes, layers, and connections needed to provide acceptable performance? To address this question, this study investigates a novel approach by merging established PINN methodologies with brain-inspired neural network techniques. We use Brain-Inspired Modular Training~(BIMT), leveraging concepts such as locality, sparsity, and modularity inspired by the organization of the brain. With brain-inspired PINN, we demonstrate the evolution of PINN architectures from large, fully connected structures to bare-minimum, compact MLP architectures, often consisting of a few neural units! Moreover, using brain-inspired PINN, we showcase the spectral bias phenomenon occurring on the PINN architectures: bare-minimum architectures solving problems with high-frequency components require more neural units than PINN solving low-frequency problems. Finally, we derive basic PINN building blocks through BIMT training on simple problems akin to convolutional and attention modules in deep neural networks, enabling the construction of modular PINN architectures. Our experiments show that brain-inspired PINN training leads to PINN architectures that minimize the computing and memory resources yet provide accurate results.

Figures (5)

• Figure 1: A graphical illustration of the contributions of this work: i) Brain-Inspired Modular Training (BIMT) liu2024seeing allows us to obtain bare-minimum PINN architectures ii) we use bare-minimum architectures as basic modules to be combined to build compact modular PINN architectures.
• Figure 2: Evolution of the brain-inspired PINN network architecture during the training for the solution of $d^2 x(t) / dt^2 = \sin(t) + 4 \sin(2t) + 9 \sin(3t) + 16 \sin(4t)$ with $x(0) = 0$ and $x(2 \pi) = 0$. The PINN architecture evolves from being fully connected to being highly sparse and modular. The red and blue lines represent connections associated with positive and negative weights. The training occurs in three phases where the strength of the L1 penalty (related to the importance of locality) changes.
• Figure 3: Analytical and brain-inspired solutions of the differential equation: $d^2 x(t) / dt^2 = \sin(t) + 4 s\in(2t) + 9 \sin(3t) + 16 \sin(4t)$ with $x(0) = 0$ and $x(2 \pi) = 0$.
• Figure 4: Bare-minimum architectures derived running brain-inspired PINN for solving basic differential equations. Interestingly, some differential equations, such as the logistic equation, only require a neural unit in the hidden layer. Another important point is that source terms with higher frequencies result in denser PINN architectures as a manifestation of the spectral bias phenomenon.
• Figure 5: Performance comparison between a fully connected MLP network with a hidden layer with nine neural units and a modular PINN using a network primitive obtained by solving $d^2 x(t) / dt^2 = \sin(t)$. The modular PINN has three basic modules with three neural units. While the fully connected and modular PINNs have similar test loss values, the MSE against the analytical solution is two orders lower for the modular PINN.