Benign overfitting in Fixed Dimension via Physics-Informed Learning with Smooth Inductive Bias

TL;DR

The paper studies kernel-based methods for linear inverse problems governed by elliptic PDEs in fixed dimensions. It derives an explicit ridge solution under spectrally transformed kernels and provides non-asymptotic excess-risk bounds that cover both regularized regression and min-norm interpolation, showing the PDE operator can stabilize variance and induce benign overfitting. A key finding is that, under a smooth inductive-bias regime, the convergence rate depends on the inverse problem and target smoothness rather than the specific low-frequency bias, with a smoothness threshold aligning with Bayesian results. The work also offers practical guidance on selecting inductive biases and demonstrates, through experiments on Poisson problems, that physics-informed interpolation can generalize well in noisy settings, informing PINN design and Sobolev-regularized learning for PDEs.

Abstract

Recent advances in machine learning have inspired a surge of research into reconstructing specific quantities of interest from measurements that comply with certain physical laws. These efforts focus on inverse problems that are governed by partial differential equations (PDEs). In this work, we develop an asymptotic Sobolev norm learning curve for kernel ridge(less) regression when addressing (elliptical) linear inverse problems. Our results show that the PDE operators in the inverse problem can stabilize the variance and even behave benign overfitting for fixed-dimensional problems, exhibiting different behaviors from regression problems. Besides, our investigation also demonstrates the impact of various inductive biases introduced by minimizing different Sobolev norms as a form of implicit regularization. For the regularized least squares estimator, we find that all considered inductive biases can achieve the optimal convergence rate, provided the regularization parameter is appropriately chosen. The convergence rate is actually independent to the choice of (smooth enough) inductive bias for both ridge and ridgeless regression. Surprisingly, our smoothness requirement recovered the condition found in Bayesian setting and extend the conclusion to the minimum norm interpolation estimators.

Figures (2)

• Figure 1: We verified our finding beyond kernel estimators. For all the plotted figure, we learn two dimensional Poisson equation. (Left) We examine the impact of smooth inductive bias on convergence. Our findings demonstrate that when the activation function is sufficiently smooth, the inductive bias has a limited effect on improving convergence, which aligns with our theoretical predictions. (Middle) Noise profile of Physics-informed interpolator and regression Interpolator. The physics-informed interpolator exhibits benign overfitting, unlike the regression interpolator. (Right) Visualization of the ground truth and the learned solutions for $f$ and $u = \Delta f$. The learned solution for $f$ effectively smooths out the high-frequency components in the error of $\Delta f$.
• Figure 2: We again verified our findings using PDE with solution of low regularity at the origin. The noise profile of Physics-informed interpolator exhibits benign overfitting, unlike the regression interpolator.

Theorems & Definitions (72)

• Definition 2.1: Sobolev Norm
• Remark 1
• Example 1: Schrödinger equation on a Hypercube
• Lemma 3.1
• proof
• Theorem 3.2: Eigenspectrum of spectrally transformed kernel $\tilde{K}$
• Remark 2
• Definition 3.3: Concentration Coefficient $\rho_{n,k}$
• Remark 3
• Theorem 3.6: Bound on Variance