Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Physics-Informed Neural Networks with Fourier Features and Attention-Driven Decoding

Rohan Arni, Carlos Blanco

TL;DR

This work introduces the Spectral PINNsformer (S-Pformer), a decoder-only transformer that replaces the encoder with Fourier feature embeddings to address spectral bias and reduce parameter count in physics-informed neural networks. By leveraging Fourier embeddings and NTK-based dynamic loss weighting, the approach achieves competitive or superior accuracy across elliptic, parabolic, and hyperbolic PDEs with fewer parameters than prior PINN transformers. Empirical results on convection, 1D-reaction, 1D-wave, and Navier–Stokes problems demonstrate substantial parameter savings (≈18.6% versus the original PINNsformer) and improved frequency-domain performance, especially in high-frequency regimes. The findings suggest that architectural simplification combined with spectral-aware encoding can enhance the efficiency and robustness of PINNs for complex spatiotemporal dynamics, with implications for scalable physics-based modeling.

Abstract

Physics-Informed Neural Networks (PINNs) are a useful framework for approximating partial differential equation solutions using deep learning methods. In this paper, we propose a principled redesign of the PINNsformer, a Transformer-based PINN architecture. We present the Spectral PINNSformer (S-Pformer), a refinement of encoder-decoder PINNSformers that addresses two key issues; 1. the redundancy (i.e. increased parameter count) of the encoder, and 2. the mitigation of spectral bias. We find that the encoder is unnecessary for capturing spatiotemporal correlations when relying solely on self-attention, thereby reducing parameter count. Further, we integrate Fourier feature embeddings to explicitly mitigate spectral bias, enabling adaptive encoding of multiscale behaviors in the frequency domain. Our model outperforms encoder-decoder PINNSformer architectures across all benchmarks, achieving or outperforming MLP performance while reducing parameter count significantly.

Physics-Informed Neural Networks with Fourier Features and Attention-Driven Decoding

TL;DR

This work introduces the Spectral PINNsformer (S-Pformer), a decoder-only transformer that replaces the encoder with Fourier feature embeddings to address spectral bias and reduce parameter count in physics-informed neural networks. By leveraging Fourier embeddings and NTK-based dynamic loss weighting, the approach achieves competitive or superior accuracy across elliptic, parabolic, and hyperbolic PDEs with fewer parameters than prior PINN transformers. Empirical results on convection, 1D-reaction, 1D-wave, and Navier–Stokes problems demonstrate substantial parameter savings (≈18.6% versus the original PINNsformer) and improved frequency-domain performance, especially in high-frequency regimes. The findings suggest that architectural simplification combined with spectral-aware encoding can enhance the efficiency and robustness of PINNs for complex spatiotemporal dynamics, with implications for scalable physics-based modeling.

Abstract

Physics-Informed Neural Networks (PINNs) are a useful framework for approximating partial differential equation solutions using deep learning methods. In this paper, we propose a principled redesign of the PINNsformer, a Transformer-based PINN architecture. We present the Spectral PINNSformer (S-Pformer), a refinement of encoder-decoder PINNSformers that addresses two key issues; 1. the redundancy (i.e. increased parameter count) of the encoder, and 2. the mitigation of spectral bias. We find that the encoder is unnecessary for capturing spatiotemporal correlations when relying solely on self-attention, thereby reducing parameter count. Further, we integrate Fourier feature embeddings to explicitly mitigate spectral bias, enabling adaptive encoding of multiscale behaviors in the frequency domain. Our model outperforms encoder-decoder PINNSformer architectures across all benchmarks, achieving or outperforming MLP performance while reducing parameter count significantly.

Paper Structure

This paper contains 36 sections, 23 equations, 10 figures, 7 tables.

Table of Contents

  1. Introduction
  2. Methodology
  3. Background
  4. Previous Architecture
  5. Pseudo-Sequence Generator
  6. Encoder-Decoder Architecture
  7. Wavelet Activation Function
  8. Spectral Architecture
  9. Embeddings
  10. Attention-Driven Decoder
  11. Linear Output Network
  12. NTK Learning Scheme
  13. Experiments
  14. Setup
  15. Model Ablation
  16. ...and 21 more sections

Figures (10)

  • Figure 1: Spectral PINNsformer Architecture
  • Figure 2: In the figure above, the first column shows the S-Pformer prediction, the middle column shows the ground truth, and the last column shows the prediction error.
  • Figure :
  • Figure :
  • Figure :
  • ...and 5 more figures