Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Finetuning greedy kernel models by exchange algorithms

Tizian Wenzel, Armin Iske

TL;DR

The paper tackles the challenge of creating sparse yet accurate kernel surrogates by marrying greedy center selection with a kernel exchange procedure. It introduces the Kernel Exchange Algorithm (KEA), which iteratively swaps centers using residual- or power-function-based criteria while keeping the expansion size fixed, leveraging the kernel representer theorem and the Newton basis for efficient updates. Across low- and high-dimensional experiments with Matérn kernels and a two-layer kernel, KEA yields substantial accuracy gains—up to $86.4\%$ improvement in some cases and an average of $17.2\%$—while maintaining the same model size. This makes KEA a practically attractive enhancement for surrogate modeling and high-dimensional kernel approximations.

Abstract

Kernel based approximation offers versatile tools for high-dimensional approximation, which can especially be leveraged for surrogate modeling. For this purpose, both "knot insertion" and "knot removal" approaches aim at choosing a suitable subset of the data, in order to obtain a sparse but nevertheless accurate kernel model. In the present work, focussing on kernel based interpolation, we aim at combining these two approaches to further improve the accuracy of kernel models, without increasing the computational complexity of the final kernel model. For this, we introduce a class of kernel exchange algorithms (KEA). The resulting KEA algorithm can be used for finetuning greedy kernel surrogate models, allowing for an reduction of the error up to 86.4% (17.2% on average) in our experiments.

Finetuning greedy kernel models by exchange algorithms

TL;DR

The paper tackles the challenge of creating sparse yet accurate kernel surrogates by marrying greedy center selection with a kernel exchange procedure. It introduces the Kernel Exchange Algorithm (KEA), which iteratively swaps centers using residual- or power-function-based criteria while keeping the expansion size fixed, leveraging the kernel representer theorem and the Newton basis for efficient updates. Across low- and high-dimensional experiments with Matérn kernels and a two-layer kernel, KEA yields substantial accuracy gains—up to improvement in some cases and an average of —while maintaining the same model size. This makes KEA a practically attractive enhancement for surrogate modeling and high-dimensional kernel approximations.

Abstract

Kernel based approximation offers versatile tools for high-dimensional approximation, which can especially be leveraged for surrogate modeling. For this purpose, both "knot insertion" and "knot removal" approaches aim at choosing a suitable subset of the data, in order to obtain a sparse but nevertheless accurate kernel model. In the present work, focussing on kernel based interpolation, we aim at combining these two approaches to further improve the accuracy of kernel models, without increasing the computational complexity of the final kernel model. For this, we introduce a class of kernel exchange algorithms (KEA). The resulting KEA algorithm can be used for finetuning greedy kernel surrogate models, allowing for an reduction of the error up to 86.4% (17.2% on average) in our experiments.
Paper Structure (10 sections, 11 equations, 3 figures, 1 algorithm)

This paper contains 10 sections, 11 equations, 3 figures, 1 algorithm.

Table of Contents

  1. Introduction
  2. Background on greedy kernel models
  3. Greedy point insertion
  4. Greedy point removal
  5. Kernel exchange algorithm: KEA
  6. Numerical experiments
  7. Greedy insertion vs. greedy removal
  8. Function approximation: Low dimensional example
  9. Function approximation: High dimensional example
  10. Conclusion & Outlook

Figures (3)

  • Figure 1: Visualization of the training error ($y$-axis) over the number of used interpolation points ($x$-axis) for the target functions in Eq. \ref{['eq:two_target_functions']}. The insertion algorithm of \ref{['subsec:greedy_insertion']} increases the number of points (operates from "left to right" on the $x$-axis), the removal algorithm of \ref{['subsec:greedy_removal']} decreases the number of points (operates from "right to left"). Both approaches yield approximately the same interpolation errors for all expansion sizes $n$.
  • Figure 2: Visualization of the improvement ratio $\Vert f - s_{n, \text{KEA}} \Vert_{L^\infty(\Omega)} / \Vert f - s_{n} \Vert_{L^\infty(\Omega)}$ ($y$-axis) over the kernel model expansion size $n$ ($x$-axis) for the four two-dimensional test functions from \ref{['subsec:low_dim_example']}: For values in $(0, 1)$, KEA yields improvements; for values in $(1, \infty)$, KEA yields deterioration. Five Matérn kernels with different smoothness parameters $p \in \{0, 1, 2, 3, 4 \}$ were used.
  • Figure 3: Visualization of the improvement ratio $\Vert f - s_{n, \text{KEA}} \Vert_{L^\infty(\Omega)} / \Vert f - s_{n} \Vert_{L^\infty(\Omega)}$ ($y$-axis) over the kernel model expansion size $n$ ($x$-axis) for the two high-dimensional test functions from \ref{['subsec:high_dim_example']}: For values in $(0, 1)$, KEA yields improvements; for values in $(1, \infty)$, KEA yields deterioration. Five Matérn kernels with different smoothness parameters $p \in \{0, 1, 2, 3, 4 \}$ were used.