The phase diagram of kernel interpolation in large dimensions
Haobo Zhang, Weihao Lu, Qian Lin
TL;DR
This work analyzes kernel interpolation in the high-dimensional regime where $n \asymp d^{\gamma}$, focusing on the inner product kernel on the sphere. By exploiting the explicit spherical-harmonic eigenstructure and the RKHS interpolation spaces $[\mathcal{H}]^{s}$, it derives exact upper and lower bounds for the variance and bias of the minimum-norm interpolant and translates them into a complete $(s,\gamma)$-phase diagram. The variance scales as $\sigma^{2} \Theta_{\mathbb P}(d^{l-\gamma} + d^{\gamma-l-1})$ and the bias as $\Theta_{\mathbb P}(d^{-(l+1)s} + d^{(2-\tilde s)l-2\gamma})$, with $l=\lfloor \gamma \rfloor$ and $\tilde s = \min\{s,2\}$, yielding a total generalization error that underpins regions of minimax optimality, sub-optimality, and inconsistency. A sharp minimax lower bound aligns with the upper bounds, establishing the first complete phase diagram for kernel interpolation in large dimensions and clarifying when benign overfitting can occur. The results have implications for understanding kernel-based generalization in highly parameterized regimes and guide design choices in high-dimensional learning settings.
Abstract
The generalization ability of kernel interpolation in large dimensions (i.e., $n \asymp d^γ$ for some $γ>0$) might be one of the most interesting problems in the recent renaissance of kernel regression, since it may help us understand the 'benign overfitting phenomenon' reported in the neural networks literature. Focusing on the inner product kernel on the sphere, we fully characterized the exact order of both the variance and bias of large-dimensional kernel interpolation under various source conditions $s\geq 0$. Consequently, we obtained the $(s,γ)$-phase diagram of large-dimensional kernel interpolation, i.e., we determined the regions in $(s,γ)$-plane where the kernel interpolation is minimax optimal, sub-optimal and inconsistent.