Iteratively reweighted kernel machines efficiently learn sparse functions

TL;DR

This work shows that kernel methods, when combined with iteratively reweighted training, can emulate neural-network-like feature learning and hierarchical representation. By coupling empirical coordinate weights with a derivative-norm (DN) estimator, the proposed IRKM algorithm detects influential coordinates and retrains kernel predictors on reweighted data, enabling efficient learning of sparse, multi-index-type functions. The authors prove gradient-consistency results for Gaussian and hypercube data and demonstrate that IRKM can learn functions with leap complexity p+1 using roughly $n\approx d^{p-1+\delta}$ samples, via a constant number of iterations, outperforming baseline kernels and often matching or surpassing neural networks in experiments. The findings imply that kernel methods, equipped with explicit feature-reuse and hierarchy-detecting mechanisms, can provide powerful, tangible feature-learning capabilities with practical impact on a broad range of learning tasks, including tabular data and epistasis studies. Overall, the paper advances a principled, kernel-based route to understanding and harnessing feature and hierarchical learning without relying on deep architectures.

Abstract

The impressive practical performance of neural networks is often attributed to their ability to learn low-dimensional data representations and hierarchical structure directly from data. In this work, we argue that these two phenomena are not unique to neural networks, and can be elicited from classical kernel methods. Namely, we show that the derivative of the kernel predictor can detect the influential coordinates with low sample complexity. Moreover, by iteratively using the derivatives to reweight the data and retrain kernel machines, one is able to efficiently learn hierarchical polynomials with finite leap complexity. Numerical experiments illustrate the developed theory.

Figures (10)

• Figure 1: (a) Test loss for kernel machines, neural networks and $\mathtt{IRKM}({\tfrac{1}{2}})$. (b) Comparison of $\mathtt{IRKM}(\alpha)$ performance for $\alpha = 0$ versus $\alpha = \tfrac{1}{2}$. (c,d) The empirical coordinate weights at the first step of $\mathtt{IRKM}(\alpha)$ with $\alpha=0$ and $\tfrac{1}{2}$ respectively. (e,f) The empirical coordinate weights after $T$ steps of $\mathtt{IRKM}(\alpha)$ with $\alpha=0$ and $\tfrac{1}{2}$ respectively. The samples are i.i.d. uniformly drawn from the hypercube $\{\pm 1\}^d$ with $d= 500$ and the label is $y = x_1 + x_2 +x_3 + x_1x_2x_3 + \varepsilon$ with $\varepsilon\sim \mathcal{N}(0,0.1^2)$. We use the Laplacian kernel for both kernel machine and $\mathtt{IRKM}(\alpha)$. The $\mathtt{IRKM}(\alpha)$ algorithm is run for at most $T=20$ steps. We train a fully-connected network by Adam, varying the widths and batch sizes as $\{128, 256, 512\}$, depth as $\{2,3,4,5,6\}$ and learning rate as $\{10^{-3}, 10^{-4}\}$ and report the smallest test loss. For all experiments, we report an average of $10$ independent runs.
• Figure 2: (a) Test loss for kernel machines, neural networks and $\mathtt{IRKM}(\tfrac{1}{2})$. (b) Comparison of $\mathtt{IRKM}(\alpha)$ performance for $\alpha = 0$ versus $\alpha = \tfrac{1}{2}$. (c)The empirical coordinate weights of $\mathtt{IRKM}(\tfrac{1}{2})$ at step $1-20$ in the regime $n = d^{0.6}$. (d,e) The empirical coordinate weights of $\mathtt{IRKM}(\alpha)$ at step $T$ for $\alpha = 0$ and $\tfrac{1}{2}$ respectively. The samples are i.i.d. uniformly drawn from the hypercube ${\{- 1,1\}^d}$ with $d= 500$ and the label is $y = x_1 + x_2 + x_1x_2x_3 + x_1x_2x_3x_4 + \varepsilon$ with $\varepsilon\sim \mathcal{N}(0,0.1^2)$. We use Laplacian kernel for both kernel machines and $\mathtt{IRKM}(\alpha)$. The $\mathtt{IRKM}(\alpha)$ algorithm is iterated for at most $T=20$ steps. We train a fully-connected network by Adam, varying the widths and batch sizes as $\{128, 256, 512\}$, depth as $\{2,3,4,5,6\}$ and learning rate as $\{10^{-3}, 10^{-4}\}$ and report the smallest test loss. For all experiments, we report an average of $10$ runs.
• Figure 3: (a) Test loss for kernel machines, neural networks and $\mathtt{RFM}(\tfrac{1}{2})$ (Algo. \ref{['alg:fullrfm']}). (b) Comparison of $\mathtt{RFM}(\alpha)$ performance for $\alpha = 0$ versus $\alpha = \tfrac{1}{2}$. (c) Eigenvalues of AGOP at step $1-50$ in the regime $n = d^{1.4}$. (d) Relative AGOP error and principal angle between the top-4 eigenspace of AGOPs. The samples are i.i.d. uniformly drawn from ${\{-1, 1\}^d}$ with $d= 500$ and label is $y = z_1 + z_2 + z_1z_2z_3 + z_1z_2z_3z_4+ \varepsilon$ with $\varepsilon\sim \mathcal{N}(0,0.1^2)$ and $z = Ux$ where $U$ is a random rotation matrix. We use the Laplacian kernel for both kernel machines and $\mathtt{RFM}(\alpha)$. We train a fully-connected network by Adam, varying the widths and batch sizes as $\{128, 256, 512\}$, depth as $\{2,3,4,5,6\}$ and learning rate as $\{10^{-3}, 10^{-4}\}$ and report the smallest test loss. For all experiments, we report an average of $10$ runs.
• Figure 4: Illustration of the effective degree of a feature as the product of learned and unlearned coordinates.
• Figure 5: Best test performance of neural networks across various widths and depths. Columns (a), (b) and (c) correspond to the experimental setting in Figure \ref{['fig:task_1']}, \ref{['fig:task_2']} and \ref{['fig:task_3']} respectively. We train a fully-connected network by Adam, varying the widths and batch sizes as $\{128, 256, 512\}$, depth as $\{2,3,4,5,6\}$ and learning rate as $\{10^{-3}, 10^{-4}\}$ and report the smallest test loss. For all experiments, we report an average of $5$ runs.

Theorems & Definitions (111)

• Theorem 1: Informal
• Theorem 2: Informal
• Theorem 3: Informal
• Theorem 4: Informal
• Theorem 5: Gradient consistency with Gaussian data
• Definition 1: Influential coordinates
• Theorem 6: Gradient consistency on hypercube
• Theorem 7: Derivative norm estimator on hypercube
• Corollary 8
• Definition 2: Leap complexity