Fast Robust Kernel Regression through Sign Gradient Descent with Early Stopping

TL;DR

This work tackles robust and sparse kernel regression by reframing kernel ridge regression (KRR) into an equivalent objective that permits switching the standard $\ell_2$ penalty to $\ell_\infty$ or $\ell_1$ penalties. It connects these explicit regularizations to early-stopping versions of gradient-based methods (kernel gradient descent, sign gradient descent, and coordinate descent), yielding computationally efficient algorithms that preserve predictive accuracy. The authors derive theoretical bounds comparing gradient flow with ridge regression, and establish loss-function correspondences that justify the use of early stopping to realize robust and sparse solutions. Empirically, robust kernel regression via sign gradient descent (KSGD) achieves similar or better performance than existing methods while being 1–2 orders of magnitude faster on real datasets, demonstrating practical scalability for large-scale kernel learning problems.

Abstract

Kernel ridge regression, KRR, is a generalization of linear ridge regression that is non-linear in the data, but linear in the model parameters. Here, we introduce an equivalent formulation of the objective function of KRR, which opens up for replacing the ridge penalty with the $\ell_\infty$ and $\ell_1$ penalties. Using the $\ell_\infty$ and $\ell_1$ penalties, we obtain robust and sparse kernel regression, respectively. We study the similarities between explicitly regularized kernel regression and the solutions obtained by early stopping of iterative gradient-based methods, where we connect $\ell_\infty$ regularization to sign gradient descent, $\ell_1$ regularization to forward stagewise regression (also known as coordinate descent), and $\ell_2$ regularization to gradient descent, and, in the last case, theoretically bound for the differences. We exploit the close relations between $\ell_\infty$ regularization and sign gradient descent, and between $\ell_1$ regularization and coordinate descent to propose computationally efficient methods for robust and sparse kernel regression. We finally compare robust kernel regression through sign gradient descent to existing methods for robust kernel regression on five real data sets, demonstrating that our method is one to two orders of magnitude faster, without compromised accuracy.

Figures (4)

• Figure 1: Comparisons of the effects of KGF/KRR, KSGD/K$\ell_\infty$R, and KCD/K$\ell_1$R on $\bm{\hat{f}}(\bm{x})$. In the top panel, a larger regularization, or a shorter training time, is used than in the bottom panel. For KGF/KRR, we use $t=1/\lambda$, while for the other two cases, $t$ and $\lambda$ are chosen so that the functions coincide as well as possible. As proposed by Proposition \ref{['thm:kgf_krr_diff']}, the KGF and KRR solutions differ most where the non-regularized solution is large, and as suggested by Proposition \ref{['thm:kgf_krr_y']}, the KGF solution tends to lie closer to the observations than the KRR solution does. For KSGD/K$\ell_\infty$R, the more extreme observations tend to be penalized harder, resulting in a more robust solution that is less sensitive to outliers. For KCD/K$\ell_1$R, some observations do not contribute to the solution, resulting in peaks at the more significant observations. The solutions obtained through early stopping are very similar to those obtained through explicit regularization, although not exactly identical.
• Figure 2: Comparisons of the effects of KGF/KRR, KSGD/K$\ell_\infty$R, and KCD/K$\ell_1$R on the seven elements in $\bm{\hat{\alpha}}$. In the top panel, a larger regularization, or a shorter training time, is used than in the bottom panel. For KGF/KRR, we use $t=1/\lambda$, while, for the two other cases, $t$ and $\lambda$ are chosen so that the functions coincide as well as possible. For KSGD/K$\ell_\infty$R, the more extreme observations tend to be penalized harder, resulting in no extreme $\hat{\alpha}_i$'s. For KCD/K$\ell_1$R, some observations do not contribute to the solution, resulting in the corresponding $\hat{\alpha}_i$'s being 0. The solutions obtained through early stopping are very similar to those obtained through explicit regularization, although not exactly identical.
• Figure 3: Modelling the data generated by Equations \ref{['eq:data_sin']} (top) and \ref{['eq:data_gauss']} (bottom) using KSGD/K$\ell_\infty$R (top) or KCD/K$\ell_1$R (bottom) KGD/KRR (both panels). For the first data set, KSGD/K$\ell_\infty$R are less affected by the outliers than KGD/KRR are, and KGD is more affected by the outliers than KRR is. For the second data set, in contrast to KGD/KRR, KCD/K$\ell_1$R are able to model the peak without incorporating the noise.
• Figure 4: Comparisons of the effects of $\ell_2$, $\ell_\infty$, and $\ell_1$ regularization on $[\bm{f}^\top,\bm{f^*}{^\top}]^\top$. In the top panel, a larger regularization is used than in the bottom panel. For $\ell_\infty$ regularization, the absolute values of many predictions are exactly equal. To obtain this, compared to the non-regularized solution, some predictions are shifted away from zero, while some predictions are shifted toward zero. For $\ell_1$ regularization, many predictions are set exactly to zero, with peaks at some extreme observations.

Theorems & Definitions (32)

• Proposition 3.1
• Remark 3.1
• Remark 4.1
• Remark 4.2
• Proposition 4.1
• Proposition 4.2
• Proposition 4.3
• Remark 5.1
• Remark 5.2
• Proposition 5.1