Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Generalization Error Curves for Analytic Spectral Algorithms under Power-law Decay

Yicheng Li, Weiye Gan, Zuoqiang Shi, Qian Lin

TL;DR

This work rigorously provides a full characterization of the generalization error curves of the kernel gradient descent method (and a large class of analytic spectral algorithms) in kernel regression, to sharpen the near inconsistency of kernel interpolation and clarify the saturation effects of kernel regression algorithms with higher qualification.

Abstract

The generalization error curve of certain kernel regression method aims at determining the exact order of generalization error with various source condition, noise level and choice of the regularization parameter rather than the minimax rate. In this work, under mild assumptions, we rigorously provide a full characterization of the generalization error curves of the kernel gradient descent method (and a large class of analytic spectral algorithms) in kernel regression. Consequently, we could sharpen the near inconsistency of kernel interpolation and clarify the saturation effects of kernel regression algorithms with higher qualification, etc. Thanks to the neural tangent kernel theory, these results greatly improve our understanding of the generalization behavior of training the wide neural networks. A novel technical contribution, the analytic functional argument, might be of independent interest.

Generalization Error Curves for Analytic Spectral Algorithms under Power-law Decay

TL;DR

This work rigorously provides a full characterization of the generalization error curves of the kernel gradient descent method (and a large class of analytic spectral algorithms) in kernel regression, to sharpen the near inconsistency of kernel interpolation and clarify the saturation effects of kernel regression algorithms with higher qualification.

Abstract

The generalization error curve of certain kernel regression method aims at determining the exact order of generalization error with various source condition, noise level and choice of the regularization parameter rather than the minimax rate. In this work, under mild assumptions, we rigorously provide a full characterization of the generalization error curves of the kernel gradient descent method (and a large class of analytic spectral algorithms) in kernel regression. Consequently, we could sharpen the near inconsistency of kernel interpolation and clarify the saturation effects of kernel regression algorithms with higher qualification, etc. Thanks to the neural tangent kernel theory, these results greatly improve our understanding of the generalization behavior of training the wide neural networks. A novel technical contribution, the analytic functional argument, might be of independent interest.
Paper Structure (42 sections, 36 theorems, 230 equations, 2 figures)

This paper contains 42 sections, 36 theorems, 230 equations, 2 figures.

Table of Contents

  1. Introduction
  2. Related works
  3. Optimality of kernel methods
  4. Recent advances in kernel ridge regression
  5. Kernel regression in the high-dimensional limit
  6. Preliminaries
  7. Reproducing kernel Hilbert space
  8. Interpolation spaces
  9. Regular RKHS
  10. Spectral algorithm
  11. Notations
  12. Main results
  13. More assumptions
  14. Main theorem
  15. Discussion
  16. ...and 27 more sections

Key Result

Proposition 2.2

Under assu:EDR and assu:RegularRKHS, the embedding index is $\alpha_0 = 1/\beta$.

Figures (2)

  • Figure 1: An illustration of the filter functions $\varphi_\lambda$ and $\psi_\lambda$.
  • Figure 2: An illustration of the contour $\Gamma_\lambda$ defined in eq:contour. The region enclosed by $\Gamma_{\lambda}$ is just $D_\lambda$ in assu:Filter. The dashed interval $[0,\kappa^2]$ contains the spectrum of $T$ and $T_X$.

Theorems & Definitions (74)

  • Remark 2.1
  • Example 2.1: Shift-invariant periodic kernels
  • Example 2.2: Dot-product kernel on the sphere
  • Example 2.3: Dot-product kernel on the ball
  • Proposition 2.2
  • Definition 2.3: Filter functions
  • Remark 2.4
  • Remark 2.5
  • Example 2.4: Kernel ridge regression
  • Example 2.5: Iterated ridge regression
  • ...and 64 more