Kernel Learning with Adversarial Features: Numerical Efficiency and Adaptive Regularization

TL;DR

The paper addresses adversarial robustness for nonparametric regression in RKHS by moving perturbations from inputs to the feature space, where the inner maximization admits a closed-form, enabling a convex reformulation that resembles ridge regression with adaptive regularization $ (|y_i - f(\boldsymbol{x}_i)| + \delta \|f\|_{\mathcal{H}})^2 $. It then develops Iterative Kernel Ridge Regression (IKRR) with the eta-trick to solve the reformulated objective efficiently, and provides generalization bounds showing adaptivity to noise without knowledge of the noise level $\sigma$, along with extensions to Adversarial Multiple Kernel Learning (AMKL). The method demonstrates competitive performance in both clean and adversarial settings, with robustness that transfers beyond the feature-perturbed objective, and offers scalable pathways via conjugate gradient and Nyström approximations. The framework connects robust kernel regression, adversarial training, and MKL, and lays groundwork for extensions to classification and infinite-width neural networks through NTK analysis.

Abstract

Adversarial training has emerged as a key technique to enhance model robustness against adversarial input perturbations. Many of the existing methods rely on computationally expensive min-max problems that limit their application in practice. We propose a novel formulation of adversarial training in reproducing kernel Hilbert spaces, shifting from input to feature-space perturbations. This reformulation enables the exact solution of inner maximization and efficient optimization. It also provides a regularized estimator that naturally adapts to the noise level and the smoothness of the underlying function. We establish conditions under which the feature-perturbed formulation is a relaxation of the original problem and propose an efficient optimization algorithm based on iterative kernel ridge regression. We provide generalization bounds that help to understand the properties of the method. We also extend the formulation to multiple kernel learning. Empirical evaluation shows good performance in both clean and adversarial settings.

Figures (6)

• Figure 1: We compare adversarial kernel training with cross-validated kernel ridge regression (Ridge CV). All experiments make use of the Matern-$5/2$ kernel. Left: we show how it fits a smooth target function. Middle: a non-smooth target function. Right: we show the test mean square error (MSE) vs the size of the training dataset $n$, and illustrate how adversarial kernel training adapts to the target function similarly to kernel ridge regression with cross validation, without the need of tuning on hyperparameters.
• Figure 2: Kernel ridge vs adversarial kernel training. Compared using $R^2$ (higher is better).
• Figure 3: We compare adversarial kernel training with cross-validated kernel ridge regression (Ridge CV). We show the test mean square error (MSE) vs training size $n$, for different kernels.
• Figure S.1: We compare adversarial kernel training with cross-validated kernel ridge regression (Ridge CV). For (a)-(c), we show how it fits a smooth target function. For (d)-(f), we show how it fits a non-smooth target function.
• Figure S.2: We compare Adversarial Kernel Training (Adv Kern) against two baseline methods: Kernel Ridge Regression with cross-validation (Ridge CV) and Kernel Ridge Regression without cross-validation (Ridge). Our evaluation includes the sine and square wave examples shown in \ref{['fig:1dexamples']}, using a Gaussian kernel, as well as a synthetic linear dataset using a linear kernel. To assess robustness to noise, we vary the standard deviation of Gaussian noise added to the data from $10^{-7}$ to $10^{0}$.

Theorems & Definitions (14)

• proposition 1
• proposition 2: closed formula for inner-maximization
• theorem 1: Excess risk for adversarial kernel training
• theorem 2: Excess risk for kernel ridge regression
• proposition 3
• proposition 4
• proof
• proposition 5
• proposition 6
• proof