On the Intrinsic Dimensions of Data in Kernel Learning

TL;DR

The results show that, for kernels such as the Laplace kernel, the effective dimension $d_K$ can be significantly smaller than the Minkowski dimension $d_\rho$, even though $d_K = d_\rho$ provably holds on regular domains.

Abstract

The manifold hypothesis suggests that the generalization performance of machine learning methods improves significantly when the intrinsic dimension of the input distribution's support is low. In the context of KRR, we investigate two alternative notions of intrinsic dimension. The first, denoted $d_ρ$, is the upper Minkowski dimension defined with respect to the canonical metric induced by a kernel function $K$ on a domain $Ω$. The second, denoted $d_K$, is the effective dimension, derived from the decay rate of Kolmogorov $n$-widths associated with $K$ on $Ω$. Given a probability measure $μ$ on $Ω$, we analyze the relationship between these $n$-widths and eigenvalues of the integral operator $φ\to \int_ΩK(\cdot,x)φ(x)dμ(x)$. We show that, for a fixed domain $Ω$, the Kolmogorov $n$-widths characterize the worst-case eigenvalue decay across all probability measures $μ$ supported on $Ω$. These eigenvalues are central to understanding the generalization behavior of constrained KRR, enabling us to derive an excess error bound of order $O(n^{-\frac{2+d_K}{2+2d_K} + ε})$ for any $ε> 0$, when the training set size $n$ is large. We also propose an algorithm that estimates upper bounds on the $n$-widths using only a finite sample from $μ$. For distributions close to uniform, we prove that $ε$-accurate upper bounds on all $n$-widths can be computed with high probability using at most $O\left(ε^{-d_ρ}\log\frac{1}ε\right)$ samples, with fewer required for small $n$. Finally, we compute the effective dimension $d_K$ for various fractal sets and present additional numerical experiments. Our results show that, for kernels such as the Laplace kernel, the effective dimension $d_K$ can be significantly smaller than the Minkowski dimension $d_ρ$, even though $d_K = d_ρ$ provably holds on regular domains.

Figures (6)

• Figure 1: Plot of $-\log(w_n)$ versus $\log n$, where $w_n$ is the output of Algorithm \ref{['empirical-width']} for the Laplace kernel on various fractal sets. Black vertical bars represent uncertainty intervals of the form $[-\log(w_n + \varepsilon), -\log(w_n)]$, where $\varepsilon$ is the maximum possible deviation from the true value of $w_K(n)$, estimated using Theorem \ref{['deterministic']} and known geometric properties of each fractal. For the Lorenz attractor, $\varepsilon$ is not reliably estimable. The slope of the fitted line, computed using the RANSAC algorithm FISCHLER1987726, is defined as inversely proportional to the empirically estimated effective dimension $d_K^{\rm emp}$.
• Figure 2: Scatter plots of $\log$-excess risk against $\log$-sample size, with fitted linear regression lines, for $d = 2,3,4$.
• Figure 3: Plot of $-\log(w^L_n)$ (and $-\log(w_n)$) versus $\log n$, where $w^L_n=\sqrt{\sum_{i>n}\hat{\lambda}_i}$ for the Laplace kernel on various fractal sets. The slope of the fitted line is computed using standard linear regression. Estimated effective dimension is defined as inversely proportional to the slope.
• Figure 4: The effective dimension and the empirical upper bound on the effective dimension for exponential type kernels $e^{-\|x-y\|^a}$ on $\Omega={\mathbb S}^{d-1}$ : $a=\frac{1}{2}$, $a=1$ (the Laplace kernel), $a=2$ (the Gaussian kernel). The first row displays plots computed with the finite sample implementation of Algorithm \ref{['empirical-width']}, whereas the second row displays plots computed using the L-BFGS--based implementation.
• Figure 5: Empirical effective dimensions for infinite-width NNGP and NTK kernels with various activation functions, computed using the L-BFGS–based algorithm. For NNGP ReLU, we have $d_K = \frac{2}{3}(d-1)$, as Table \ref{['rates']} indicates. We do not show plots for the Leaky ReLU activation function ($\alpha<1$) due to their similarity to the ReLU case.

Theorems & Definitions (42)

• Theorem 1
• proof
• Theorem 2
• Theorem 3
• Remark 1
• Theorem 4: Ismagilov1968
• proof : Proof of the first part of Theorem \ref{['from-isma']}
• Lemma 1
• Theorem 5
• Lemma 2