Adaptive Kernel Methods

TL;DR

The paper introduces adaptive kernel methods that replace a fixed ambient RKHS with parameterized, learnable RKHSs to form $f^{\Lambda}=\mathcal{P}^{\Lambda}_{S,E}F$ and optimize both the projection and the basis. It develops finite-dimensional, parameter-dependent kernels $\xi_D^{\Lambda}$ with provable error bounds, and provides a concrete analysis in $H_2(\mathbb{D})$ using Takenaka-Malmquist and Laguerre bases. Training is cast as a joint optimization over $\Lambda$ and weights within a $D$-dimensional RKHS, with existence guaranteed under standard continuity/compactness assumptions (via Berge’s maximum theorem). The framework unifies and extends Random Fourier Features, showing RFF as a special case while enabling gradient-based refinement of the basis to improve accuracy and efficiency. Empirical results on LTI system identification and forest cover classification demonstrate that adaptive kernels can achieve higher accuracy with fewer parameters and comparable training cost, highlighting their practical potential for large-scale nonlinear mapping tasks.

Abstract

Kernel methods approximate nonlinear maps in a data-driven manner by projecting the target map onto a finite-dimensional Hilbert space called the solution space. Traditionally, this space is a subspace of a fixed ambient reproducing kernel Hilbert space (RKHS), determined solely by the chosen kernel and the dataset, whose elements identify the basis elements. Consequently, the projection operator underlying the kernel method depends on the loss function, the dataset, and the choice of ambient RKHS. In this study, we consider kernel methods whose solution spaces also depend on learnable parameters that are independent of the dataset. The resulting methods can be viewed as variable projection operators that depend on the loss function, the dataset, and the new learnable parameters instead of a fixed RKHS. This work has two main contributions. First, we propose an efficient approximation of kernels associated with infinite-dimensional RKHSs, commonly used to reduce the solution-space dimension for large datasets. Second, we construct fixed-dimensional, parameter-dependent solution spaces that enable highly efficient kernel models suitable for large-scale problems without the need to approximate kernels of infinite-dimensional RKHSs. Our novel family of adaptive kernel methods generalizes earlier approaches, including Random Fourier Features, and we demonstrate their effectiveness through several numerical experiments.

Figures (2)

• Figure 1: True and reconstructed frequency response $H_{|\mathbb{T}}$ using trigonometric and TM based adaptive kernel models. TOP: absolute value of frequency response. BOTTOM: phase of frequency response.
• Figure 2: True nonlinear parameters (mirror image poles) of the transfer function $H$ and the learned parameters of the TM based adaptive kernel model.

Theorems & Definitions (24)

• Definition 2.1: Reproducing kernel Hilbert space
• Definition 2.2
• Theorem 2.1: Properties of reproducing kernels
• Theorem 2.2: Density of kernel slices
• proof
• Theorem 2.3: Representer theorem
• Theorem 2.4: Pointwise convergence theorem
• Theorem 2.5: Mercer's theorem
• Definition 2.3: Feature map, feature space
• Theorem 2.6: Optimal kernel in nested RKHSs