REAct: Rational Exponential Activation for Better Learning and Generalization in PINNs
Sourav Mishra, Shreya Hallikeri, Suresh Sundaram
TL;DR
This work addresses optimization and generalization challenges in PINNs arising from activation function limitations. It introduces Rational Exponential Activation (REAct), a four-parameter, tanh-like activation with the form $REAct(x) = \frac{1 - \exp(a x + b)}{1 + \exp(c x + d)}$, designed to provide smooth, unbounded outputs with tunable zero-crossings and curvature. Across five forward IBVPs, function-approximation tasks, and inverse problems, REAct outperforms standard activations and ABU-PINN, achieving three orders of magnitude lower MSE on a heat problem and demonstrating robust noise rejection. The results demonstrate that REAct enhances both optimization and generalization in PINNs, enabling accurate parameter estimation and improved performance on finer grids and unseen inputs.
Abstract
Physics-Informed Neural Networks (PINNs) offer a promising approach to simulating physical systems. Still, their application is limited by optimization challenges, mainly due to the lack of activation functions that generalize well across several physical systems. Existing activation functions often lack such flexibility and generalization power. To address this issue, we introduce Rational Exponential Activation (REAct), a generalized form of tanh consisting of four learnable shape parameters. Experiments show that REAct outperforms many standard and benchmark activations, achieving an MSE three orders of magnitude lower than tanh on heat problems and generalizing well to finer grids and points beyond the training domain. It also excels at function approximation tasks and improves noise rejection in inverse problems, leading to more accurate parameter estimates across varying noise levels.