An Agnostic View on the Cost of Overfitting in (Kernel) Ridge Regression
Lijia Zhou, James B. Simon, Gal Vardi, Nathan Srebro
TL;DR
This work studies the cost of overfitting in noisy kernel ridge regression from an agnostic PAC standpoint, defining the cost as the ratio between the ridgeless test risk and the optimally regularized risk as a function of sample size and task structure. Using Gaussian-design risk predictions tied to the eigenstructure of the kernel, it derives a tight agnostic bound on the cost via the learnability coefficient $\mathcal{E}_0$ and the effective ranks of the covariance. The authors establish a refined taxonomy—benign, tempered, and catastrophic overfitting—by linking the spectrum geometry (through $r_k$ and $R_k$) to the asymptotic behavior of $\mathcal{E}_0$ and provide concrete finite-sample bounds. They also apply the framework to inner-product kernels in the polynomial regime, showing how a multiple-descent phenomenon arises and how the ridgeless predictor can remain competitive under appropriate spectral conditions. Overall, the paper connects closed-form risk predictions with spectrum-based notions of effective rank to deliver interpretable, agnostic guarantees for when interpolation is nearly optimal.
Abstract
We study the cost of overfitting in noisy kernel ridge regression (KRR), which we define as the ratio between the test error of the interpolating ridgeless model and the test error of the optimally-tuned model. We take an "agnostic" view in the following sense: we consider the cost as a function of sample size for any target function, even if the sample size is not large enough for consistency or the target is outside the RKHS. We analyze the cost of overfitting under a Gaussian universality ansatz using recently derived (non-rigorous) risk estimates in terms of the task eigenstructure. Our analysis provides a more refined characterization of benign, tempered and catastrophic overfitting (cf. Mallinar et al. 2022).