Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

On the optimal shape parameter for kernel methods: Sharp direct and inverse statements

Tizian Wenzel, Gabriele Santin

TL;DR

This work establishes a theoretical framework for kernel based approximation for Radial Basis Function by leveraging a recently established theory on sharp direct, inverse and saturation statements for kernel based approximation, and links the search for the optimal shape parameter to superconvergence phenomena.

Abstract

The search for the optimal shape parameter for Radial Basis Function (RBF) kernel approximation has been an outstanding research problem for decades. In this work, we establish a theoretical framework for this problem by leveraging a recently established theory on sharp direct, inverse and saturation statements for kernel based approximation. In particular, we link the search for the optimal shape parameter to superconvergence phenomena. Our analysis is carried out for finitely smooth Sobolev kernels, thereby covering large classes of radial kernels used in practice, including those emerging from current machine-learning methodologies. Our results elucidate how approximation regimes, kernel regularity, and parameter choices interact, thereby clarifying a question that has remained unresolved for decades.

On the optimal shape parameter for kernel methods: Sharp direct and inverse statements

TL;DR

This work establishes a theoretical framework for kernel based approximation for Radial Basis Function by leveraging a recently established theory on sharp direct, inverse and saturation statements for kernel based approximation, and links the search for the optimal shape parameter to superconvergence phenomena.

Abstract

The search for the optimal shape parameter for Radial Basis Function (RBF) kernel approximation has been an outstanding research problem for decades. In this work, we establish a theoretical framework for this problem by leveraging a recently established theory on sharp direct, inverse and saturation statements for kernel based approximation. In particular, we link the search for the optimal shape parameter to superconvergence phenomena. Our analysis is carried out for finitely smooth Sobolev kernels, thereby covering large classes of radial kernels used in practice, including those emerging from current machine-learning methodologies. Our results elucidate how approximation regimes, kernel regularity, and parameter choices interact, thereby clarifying a question that has remained unresolved for decades.
Paper Structure (19 sections, 7 theorems, 33 equations, 6 figures, 1 table)

This paper contains 19 sections, 7 theorems, 33 equations, 6 figures, 1 table.

Table of Contents

  1. Introduction
  2. Background
  3. Kernels, RKHS and power spaces
  4. Kernels and RKHS
  5. Geometric quantities
  6. Power spaces
  7. Sharp direct and inverse statements
  8. Related work on the optimal shape parameter
  9. The optimal shape parameter and superconvergence
  10. Non-asymptotic vs asymptotic regime
  11. Sharp direct and inverse statements
  12. Discussion of the main results
  13. Implications and Applications
  14. No optimal kernel in the escaping regime
  15. Optimal kernels in the superconvergence regime
  16. ...and 4 more sections

Key Result

Theorem 3

Under ass:kernel_domain, consider $f \in \mathcal{C}(\Omega)$ such that $f \in (\mathcal{H}_k (\Omega))_\vartheta$ for some $\vartheta \in (\frac{d}{2\tau}, 2]$ Let $\rho_0 > 0$. Then there exists a constant $c_f > 0$ such that it holds for any set of quasi-uniform points $X \subset \Omega$ with $\frac{h_X}{q_X} \leq \rho_0$ and $h_X\leq 1$.

Figures (6)

  • Figure 1: Visualization of the $\Vert f - s_{f, k, X} \Vert_{L_2(\Omega)}$ error over the shape parameter $\varepsilon$. This is a typical situation frequently reported in the literature, which will be explained by our analysis. In fact, the red curve is depicted again in the middle left plot of \ref{['fig:ex2_1d']}.
  • Figure 2: Visualization of the scale of power spaces (top arrow) and Sobolev spaces (bottom arrow). Several special cases like $L_2(\Omega)$, $\mathcal{H}_k (\Omega) \asymp H^{\tau}(\Omega)$ and $T_k L_2(\Omega) \subset H^{2\tau}(\Omega)$ are depicted, as well as the equivalence and subset relations of Eq. \ref{['eq:power_spaces_escaping']} and \ref{['eq:power_spaces_superconv']}.
  • Figure 3: Visualization of the three kernels $k^1(\cdot, x),k^2(\cdot, x), k^3(\cdot, x)$ (from top to bottom), centered at $x=1/6,1/2,5/6$, and scaled with $\varepsilon=1$ (left) and $\varepsilon=100$ (right).
  • Figure 4: Example \ref{['ex:no_dip']}: Error decay (first column) and rate of decay (second column) as a function of the shape parameter $\varepsilon$ for increasing number of points $|X|$, and for the three kernels. The first column additionally reports the interpolation error in the limit $\varepsilon\to0$ extended to all $\varepsilon$ (dotted lines). The second column shows horizontal lines for the basic rate in $\mathcal{H}_k (\Omega)$ and the full superconvergence rate, and vertical lines for the expected optimal shape parameter(s), when existing.
  • Figure 5: Example \ref{['ex:dip']}: Error decay (first column) and rate of decay (second column) as a function of $\varepsilon$ for increasing $n$, and for the three kernels. The first column additionally reports the interpolation error in the limit $\varepsilon\to0$ extended to all $\varepsilon$ (dotted lines). The second column shows horizontal lines for the basic rate in $\mathcal{H}_k (\Omega)$ and the full superconvergence rate, and vertical lines for the expected optimal shape parameter(s), when existing.
  • ...and 1 more figures

Theorems & Definitions (21)

  • Theorem 3
  • proof
  • Theorem 4
  • proof
  • Definition 5
  • Definition 6
  • Theorem 7: One-to-one correspondence
  • Definition 8
  • Example 10
  • Example 11
  • ...and 11 more