Universality of Kernel Random Matrices and Kernel Regression in the Quadratic Regime

TL;DR

The paper extends kernel ridge regression analysis beyond the commonly studied proportional regime to the quadratic regime where $n\asymp d^2$, proving that many inner-product kernels are spectrally well-approximated by a degree-2 polynomial kernel with explicit correction terms. It derives a deformed Marchenko-Pastur law as the limiting eigenvalue distribution and provides precise asymptotics for training and generalization errors under Gaussian moment-matching data with covariance structure. The results demonstrate that KRR in the quadratic regime can learn nonlinear quadratic targets and exhibit double-descent behavior, with a consistent generalized cross-validation estimator. The work advances the theoretical understanding of kernel methods in high dimensions and offers tools that can extend to higher-order polynomial regimes and more general data distributions.

Abstract

Kernel ridge regression (KRR) is a popular class of machine learning models that has become an important tool for understanding deep learning. Much of the focus thus far has been on studying the proportional asymptotic regime, $n \asymp d$, where $n$ is the number of training samples and $d$ is the dimension of the dataset. In the proportional regime, under certain conditions on the data distribution, the kernel random matrix involved in KRR exhibits behavior akin to that of a linear kernel. In this work, we extend the study of kernel regression to the quadratic asymptotic regime, where $n \asymp d^2$. In this regime, we demonstrate that a broad class of inner-product kernels exhibits behavior similar to a quadratic kernel. Specifically, we establish an operator norm approximation bound for the difference between the original kernel random matrix and a quadratic kernel random matrix with additional correction terms compared to the Taylor expansion of the kernel functions. The approximation works for general data distributions under a Gaussian-moment-matching assumption with a covariance structure. This new approximation is utilized to obtain a limiting spectral distribution of the original kernel matrix and characterize the precise asymptotic training and test errors for KRR in the quadratic regime when $n/d^2$ converges to a non-zero constant. The generalization errors are obtained for (i) a random teacher model, (ii) a deterministic teacher model where the weights are perfectly aligned with the covariance of the data. Under the random teacher model setting, we also verify that the generalized cross-validation (GCV) estimator can consistently estimate the generalization error in the quadratic regime for anisotropic data. Our proof techniques combine moment methods, Wick's formula, orthogonal polynomials, and resolvent analysis of random matrices with correlated entries.

Figures (5)

• Figure 1: Spectral distributions of $\frac{2\alpha}{f"(0)}(\boldsymbol{K}-a {\mathbf I}_n)$ for $f(x)=\cos(x)$, $n=10000$ and $d=200$, and limiting density function of \ref{['eq:defmu']} in red curves. For dataset ${\boldsymbol X}$, we use Gaussian data with population covariance: ${\boldsymbol \Sigma}={\mathbf I}_d$ and ${\boldsymbol \Sigma}= {\boldsymbol \Sigma}_0$ which is defined by \ref{['eq:Sigma_0']}.
• Figure 2: Theoretical curves of bias term $\mathcal{B}(\lambda_*)$ (green), variance term $\sigma_{\boldsymbol{\epsilon}}^2 {\mathcal{V}}(\lambda_*)$ (red), and the generalization error (yellow) from Theorem \ref{['thm:test_limit']}. We fix $d=1000$ and vary the sample size $n$. The ridge parameter $\lambda = 10^{-3},10^{-2}$ and noise level $\sigma_{\boldsymbol{\epsilon}} = 0.25$. The plot reveals a double-descent phenomenon in the quadratic regime $n \propto d^2$.
• Figure 3: Spectral distributions for kernel function $f(x)=x^2+x$ with isotropic and anisotropic Gasussian datasets. The red curves are given by the limiting spectral distribution obtained from Theorem \ref{['thm:globallaw']}. The number of outliers is $O(d)$ plotted in the subfigures, due to the low-rank terms in $\boldsymbol{K}^{(2)}$; see \ref{['eq:K2']}.
• Figure 4: Numerical simulations for the operator norm $\|\boldsymbol{K}-\boldsymbol{K}^{(2)}\|$ for exponential kernel $f(x)=\exp(x)$ when varying $d$ and fixing the ratio $\alpha = \frac{d^2}{2n} = 1.2$ and $0.8$. For each $n$ and $d$, we take 15 trials to average the error.
• Figure 5: Test losses (orange points) and theoretical prediction (blue lines) of $\mathcal{R}(\lambda)$ for different aspect ratios $\alpha$ and teacher models $f_*$. Fix $d = 160$, noise level $\sigma_{\boldsymbol{\epsilon}} = 0.5$, and ridge parameter $\lambda = 0.01$. We choose the kernel function as $f(x)=(1+x)^2$. For each simulation point, we take 8 averages. (a) The teacher model $f_*$ is defined by \ref{['eq:teacher']} with coefficients $c_0 =1,c_1=2,c_2=1$ and the theoretical curve is given by Theorem \ref{['thm:test_limit']}. (b) The teacher model $f_*$ is identical to (a) but replaces ${\boldsymbol G}$ in \ref{['eq:teacher']} with ${\mathbf I}_d$ and the theoretical curve is derived from Theorem \ref{['thm:test_limit_deterministic']}.

Theorems & Definitions (49)

• Theorem 5: Quadratic kernel approximation
• Theorem 8: Limiting eigenvalue distribution
• Theorem 11: Asymptotic training error
• Theorem 14: Asymptotic generalization error for random $f_*$
• Remark 15: Connection to double descent and multiple descent
• Remark 16
• Theorem 17: Asymptotic generalization error for deterministic $f_*$
• Remark 18
• Corollary 19
• Definition 20: Stieltjes transform