Predicting kernel regression learning curves from only raw data statistics

TL;DR

This work develops an end-to-end theory for kernel regression on realistic, anisotropic data by introducing the Hermite eigenstructure Ansatz (HEA), which predicts the kernel eigensystem from the data covariance ${\bm{\Sigma}}$ and a Hermite decomposition of the target $f_*$. By showing HEA exactness for Gaussian data in certain limits and robust empirical agreement on real image datasets, the authors enable predicting test loss curves for kernel ridge regression without constructing the kernel matrix. They demonstrate end-to-end predictivity by (i) applying HEA to CIFAR-5m, SVHN, and ImageNet tasks, (ii) introducing a robust method to recover target Hermite coefficients via Gram-Schmidt, and (iii) connecting HEA predictions to MLPs in the feature-learning regime, where optimization time scales with the predicted eigenvalues. The results suggest a principled, data-driven pathway to anticipate learning curves and guide hyperparameters in nontrivial learning settings, advancing the prospect of end-to-end theories that bridge dataset structure and model performance. Overall, the paper provides a concrete, testable framework for predicting learning dynamics from simple data statistics and target structure, with demonstrated applicability to real-world vision data and neural network regimes.

Abstract

We study kernel regression with common rotation-invariant kernels on real datasets including CIFAR-5m, SVHN, and ImageNet. We give a theoretical framework that predicts learning curves (test risk vs. sample size) from only two measurements: the empirical data covariance matrix and an empirical polynomial decomposition of the target function $f_*$. The key new idea is an analytical approximation of a kernel's eigenvalues and eigenfunctions with respect to an anisotropic data distribution. The eigenfunctions resemble Hermite polynomials of the data, so we call this approximation the Hermite eigenstructure ansatz (HEA). We prove the HEA for Gaussian data, but we find that real image data is often "Gaussian enough" for the HEA to hold well in practice, enabling us to predict learning curves by applying prior results relating kernel eigenstructure to test risk. Extending beyond kernel regression, we empirically find that MLPs in the feature-learning regime learn Hermite polynomials in the order predicted by the HEA. Our HEA framework is a proof of concept that an end-to-end theory of learning which maps dataset structure all the way to model performance is possible for nontrivial learning algorithms on real datasets.

Figures (25)

• Figure 1: We provide an end-to-end theory of learning for kernel ridge regression (KRR) that maps minimal statistics of the data distribution to test-time performance.(Top left.) KRR implicitly consists of two steps: (1) the kernel maps the data to high-dimensional nonlinear features ${\bm{x}}\mapsto{\bm{\psi}}({\bm{x}})$, then (2) it fits a linear estimator to these features. Let the covariance in data space be $\mathbb{E}_{} \! \left[ {\bm{x}}{\bm{x}}^\top \right] = {\bm{U}}{\bm{\Gamma}}{\bm{U}}^\top$ and let the covariance in feature space be $\mathbb{E}_{} \! \left[ {\bm{\psi}}({\bm{x}}){\bm{\psi}}({\bm{x}})^\top \right] = {\bm{\Xi}}{\bm{\Lambda}}{\bm{\Xi}}^\top$. (Lower left.) We introduce an ansatz (blue box) that predicts the feature covariance statistics (${\bm{\Xi}}, {\bm{\Lambda}}$) from only the data covariance (${\bm{U}}, {\bm{\Gamma}}$) and the functional form of ${\bm{\psi}}(\cdot)$. This is sufficient to predict average-case test error using known results (green box). (Top right.) We are able to predict learning curves for KRR on image tasks without requiring omniscient knowledge of the feature statistics (i.e., without ever constructing or diagonalizing a kernel matrix). (Bottom right.) We are able to accurately predict the KRR sample complexity, including constant prefactors, for learning polynomials of the ImageNet dataset. See \ref{['fig:lrn_curves']} for additional plots and \ref{['app:hea_check_expts']} for experimental details.
• Figure 2: The Hermite eigenstructure ansatz (\ref{['eqn:hea']}) accurately predicts the eigenvalues (top) and eigenfunctions (bottom) of various kernel/dataset combinations. For four kernel/dataset settings (columns), we compute the empirical kernel eigensystem $\{(\lambda^{(\text{emp})}_i, \phi^{(\text{emp})}_i)\}$ and compare to the theoretical eigenpairs $\{(\lambda^{(\text{th})}_i, \phi^{(\text{th})}_i)\}$ obtained from \ref{['def:he']} and indexed in order of decreasing $\lambda^{(\text{th})}_i$. In the top plot in each column, the $i$-th point from the top right has coordinates $(\lambda^{\text{(emp})}_i, \lambda^{\text{(th})}_i)$ and its color indicates the polynomial degree of $\phi_i^{(\text{th})}$. In the bottom plot in each column, we bin both the predicted and empirical eigenfunctions into logarithmic spectral bins and visualize the pairwise subspace overlap (\ref{['eqn:overlap']}), with axes matching the top plot. Grey pixels indicate bins with no eigenvalues. In all plots, concentration along the diagonal indicates theory-experiment match. See \ref{['asec:expt-2']} for further explanation and experimental details.
• Figure 3: Using only the empirical data covariance and a polynomial decomposition of the target function, we predict learning curves across a variety of kernels, datasets, and targets.(Top row.) We predict test error on synthetic target functions of real datasets. Each target is the multi-Hermite polynomial (\ref{['eqn:multi_hermite_def']}) whose leading term is indicated in the plots. (Recall that $z_i={\bm{u}}_i^\top{\bm{x}}/\sqrt{\gamma_i}$ are the rescaled PCA coordinates.) For each of these targets, our ansatz predicts both learning curves (top left) and the sample complexity required to achieve $\texttt{MSE}\leq0.5$ (top right). (Bottom left and center.) We train on binarized true target functions. See \ref{['asec:datasets']} for details. (Bottom right.) We construct synthetic targets by drawing from a Gaussian process. The source exponent controls the difficulty of the target. See \ref{['asec:datasets']} for details. For all learning curve predictions, we estimate the coefficients of the target function in the predicted eigenbasis using the Gram-Schmidt process described in \ref{['asec:tfdecomp']}. See \ref{['asec:lrncrv']} for full experimental details.
• Figure 4: The HEA accurately predicts polynomial learning order in MLPs in the feature-learning regime. We measure the amount of time it takes to train three-layer ReLU MLPs, each dot being a trained MLP on one multi-Hermite polynomial target once a test error of $\texttt{MSE}\leq0.1$ is reached. We find that the optimization time is well-predicted by the inverse square root of the HEA eigenvalue $\lambda_{\bm{\alpha}}^{-1/2}$.
• Figure 5: We compute the Laplace kernel on-sphere coefficients $c_\ell$ with $\sigma = r = 1$ and plot $c_\ell \gamma_1^\ell$ for various $\ell$. Dashed lines show $r^2 \gamma_1^{-1}$, the value of the predicted minimum.

Theorems & Definitions (27)

• Definition 1: Rotation-invariant kernel
• Definition 2: Dot-product kernel
• Definition 3: On-sphere level coefficients
• Definition 4: Hermite eigensystem
• Theorem 1: The HEA holds for a wide Gaussian kernel on a Gaussian measure
• Theorem 2: The HEA holds for a fast-decaying dot-product kernel on a Gaussian measure
• Theorem 3: Courant–Fischer–Weyl min-max principle
• proof
• Corollary 1: Cauchy interlacing law
• Theorem 4: Davis–Kahan $\sin \Theta$ theorem davis:1970-eigenvector-perturbation