Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Just Interpolate: Kernel "Ridgeless" Regression Can Generalize

Tengyuan Liang, Alexander Rakhlin

TL;DR

This work tackles why minimum-norm interpolating kernel methods can generalize without explicit regularization. It derives a data-dependent risk bound for ridgeless kernel interpolation in RKHS, decomposing error into variance- and bias-like terms controlled by kernel curvature and data spectral properties, and shows an implicit regularization mechanism in high dimensions. Theoretical results are complemented by MNIST experiments and synthetic simulations that demonstrate interpolation-limited performance in favorable spectral regimes and verify the predicted bias-variance trade-off. Overall, the paper connects kernel geometry, random-matrix insights, and high-dimensional statistics to explain practical success of interpolation and to guide kernel design in practice.

Abstract

In the absence of explicit regularization, Kernel "Ridgeless" Regression with nonlinear kernels has the potential to fit the training data perfectly. It has been observed empirically, however, that such interpolated solutions can still generalize well on test data. We isolate a phenomenon of implicit regularization for minimum-norm interpolated solutions which is due to a combination of high dimensionality of the input data, curvature of the kernel function, and favorable geometric properties of the data such as an eigenvalue decay of the empirical covariance and kernel matrices. In addition to deriving a data-dependent upper bound on the out-of-sample error, we present experimental evidence suggesting that the phenomenon occurs in the MNIST dataset.

Just Interpolate: Kernel "Ridgeless" Regression Can Generalize

TL;DR

This work tackles why minimum-norm interpolating kernel methods can generalize without explicit regularization. It derives a data-dependent risk bound for ridgeless kernel interpolation in RKHS, decomposing error into variance- and bias-like terms controlled by kernel curvature and data spectral properties, and shows an implicit regularization mechanism in high dimensions. Theoretical results are complemented by MNIST experiments and synthetic simulations that demonstrate interpolation-limited performance in favorable spectral regimes and verify the predicted bias-variance trade-off. Overall, the paper connects kernel geometry, random-matrix insights, and high-dimensional statistics to explain practical success of interpolation and to guide kernel design in practice.

Abstract

In the absence of explicit regularization, Kernel "Ridgeless" Regression with nonlinear kernels has the potential to fit the training data perfectly. It has been observed empirically, however, that such interpolated solutions can still generalize well on test data. We isolate a phenomenon of implicit regularization for minimum-norm interpolated solutions which is due to a combination of high dimensionality of the input data, curvature of the kernel function, and favorable geometric properties of the data such as an eigenvalue decay of the empirical covariance and kernel matrices. In addition to deriving a data-dependent upper bound on the out-of-sample error, we present experimental evidence suggesting that the phenomenon occurs in the MNIST dataset.

Paper Structure

This paper contains 21 sections, 12 theorems, 89 equations, 8 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Literature Review
  3. Preliminaries
  4. Problem Formulation
  5. Notation and Background on RKHS
  6. Main Result
  7. Discussion of the assumptions
  8. Behavior of the Data-dependent Bound
  9. Case $n>d$
  10. Case $d>n$
  11. Proofs
  12. Bias-Variance Decomposition
  13. Variance
  14. Bias
  15. Experiments
  16. ...and 6 more sections

Key Result

Theorem 1

Define Under the assumptions (A.1)-(A.4) and for $d$ large enough, with probability at least $1-2\delta-d^{-2}$ (with respect to a draw of design matrix $X$), the interpolation estimator eq:interpolation_closedform satisfies Here the remainder term $\epsilon(n, d) = O(d^{-\frac{m}{8+m}} \log^{4.1} d) + O(n^{-\frac{1}{2}} \log^{0.5} (n/\delta))$.

Figures (8)

  • Figure 1: Test performance of Kernel Ridge Regression on pairs of MNIST digits for various values of regularization parameter $\lambda$, normalized by variance of $y$ in the test set (for visualization purposes).
  • Figure 2: Generalization error as a function of varying spectral decay. Here $d = 200$, $n = 400, 1000, 2000, 4000$.
  • Figure 3: Generalization error as a function of varying spectral decay. Here $n=200$, $d = 400, 1000, 2000, 4000$.
  • Figure 4: Test error, normalized as in \ref{['eq:normalized_mse']}. The y-axis is on the log scale.
  • Figure 5: Spectral decay. The y-axis is on the log scale.
  • ...and 3 more figures

Theorems & Definitions (29)

  • Theorem 1
  • Corollary 4.1: General spectral decay: $n>d$
  • Example 4.1: Low rank
  • Example 4.2: Approx. low rank
  • Example 4.3: Nonparametric slow decay
  • Corollary 4.2: General spectral decay: $d > n$
  • Example 4.4: Favorable spectral decay for $d\gg n$
  • Lemma 5.1
  • proof : Proof of Lemma \ref{['lem:decomposition']}
  • Theorem 2: Variance
  • ...and 19 more