xLSTM-PINN: Memory-Gated Spectral Remodeling for Physics-Informed Learning

TL;DR

This work addresses spectral bias in physics-informed neural networks by introducing xLSTM-PINN, an architecture that performs representation-level spectral remodeling through memory-gated residual micro-steps. The method preserves automatic differentiation and physics-loss constraints while reshaping the neural tangent kernel to boost high-frequency learning, supported by NTK-based analysis and frequency-domain benchmarks. Across four PDE benchmarks with matched budgets, xLSTM-PINN achieves substantial reductions in spectral error and RMSE, faster convergence, and expanded resolvable bandwidth, validating both theory and practice. The results indicate a robust, architecture-level pathway to improve accuracy, reproducibility, and transferability in physics-informed learning without altering optimization or loss formulations.

Abstract

Physics-informed neural networks (PINN) face significant challenges from spectral bias, which impedes their ability to model high-frequency phenomena and limits extrapolation performance. To address this, we introduce xLSTM-PINN, a novel architecture that performs representation-level spectral remodeling through memory gating and residual micro-steps. Our method consistently achieves markedly lower spectral error and root mean square error (RMSE) across four diverse partial differential equation (PDE) benchmarks, along withhhh a broader stable learning-rate window. Frequency-domain analysis confirms that xLSTM-PINN elevates high-frequency kernel weights, shifts the resolvable bandwidth rightward, and shortens the convergence time for high-wavenumber components. Without modifying automatic differentiation or physics loss constraints, this work provides a robust pathway to suppress spectral bias, thereby improving accuracy, reproducibility, and transferability in physics-informed learning.

Figures (10)

• Figure 1: xLSTM-PINN overview and intra-block recursion. We embed coordinates by a linear–tanh map to $u^{(0)}$, apply per-layer xLSTM blocks with $S$ micro-steps, and output $u_\theta$ via a linear head; Automatic Differentiation (AD) provides PDE residuals and boundary terms for the loss. (a) Top level: $\bf{x} \to u^{(0)} \to [\text{xLSTM block + linear--tanh}]^L \to u_\theta$. (b) Inside a block: compute LSTM gating $(h, c)$, refine by residual updates for $S$ micro-steps, then pass to the next layer.
• Figure 2: Frequency-domain benchmark evidences spectral-bias suppression and bandwidth expansion of xLSTM-PINN. Under matched training budgets and model size, we probe the spectrum with plane waves and report endpoint error $E_T(|k|)$, gain $G(|k|) = E_{\text{base}} / E_{\text{xLSTM}}$, and time-to-threshold $\tau(|k|)$, with uncertainty shown as confidence bands and the resolution frontier $k^*(\varepsilon)$ indicating the learnable bandwidth. This benchmark isolates representation effects and directly quantifies improved high-frequency learnability driven by memory gating and residual micro-steps. (a) Endpoint error vs frequency: $E_T(|k|)$ decreases systematically in the mid-high range, with the error plateau lowered and extended rightward; (b) Spectral gain vs frequency: $G(|k|)$ exceeds 1 over a broad band; non-overlapping confidence intervals highlight robust, high-frequency advantages; (c) Time-to-threshold vs frequency: $\tau(|k|)$ shifts downward with a right-shifted $k^*(\varepsilon)$, demonstrating higher resolvable wavenumbers at the same budget, reduced spectral bias, and improved accuracy.
• Figure 3: 2D Laplace with mixed Dirichlet-Neumann boundaries: solution, prediction, and absolute-error comparison for xLSTM-PINN and PINN. Domain $[0,1]^2$; boundaries $\phi(x,0) = 0, \phi(x,1) = 1, \partial_x \phi(0,y) = 0, \partial_x \phi(1,y) = 0$; analytic solution $\phi^*(x,y) = y$. (a) xLSTM-PINN: isocontours align with the analytic field; the absolute-error map is deep blue with fine vertical striations, magnitude on the order of $10^{-4}$. (b) PINN: a broad smooth warm curved band spans the domain, magnitude on the order of $10^{-2}$, accumulating along $y$ and indicating a low-order shape bias.
• Figure 4: Training loss histories for the 2D Laplace mixed-boundary case under matched sampling and optimization. (a) xLSTM–PINN reaches a low-loss regime faster and sustains a lower loss floor; (b) PINN on a logarithmic axis highlights slower late-phase decay and larger oscillations.
• Figure 5: Steady heat conduction in a disk (uniform volumetric source $+$ Robin convective boundary): using the FEM field as the validation set, we compare xLSTM-PINN and PINN on solution, PDE prediction, and absolute error. Domain $\Omega = \{x^2 + y^2 \leq 1\}$; PDE $\theta_{xx} + \theta_{yy} + 1 = 0$; boundary $-\partial_n \theta = \text{Bi}\, \theta$. (a) xLSTM-PINN: concentric isotherms with a thin cool error ring near the rim. (b) PINN: a wider warm error ring penetrating inward.