Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Truncated Kernel Stochastic Gradient Descent with General Losses and Spherical Radial Basis Functions

Jinhui Bai, Andreas Christmann, Lei Shi

TL;DR

This work addresses scalable kernel-based online learning with general losses on spherical data by introducing Truncated Kernel SGD (T-kernel SGD). The method leverages spherical radial basis functions to decompose the RKHS into nested finite-dimensional spaces and projects stochastic gradients onto these spaces, enabling adaptive regularization and efficient updates. The authors prove minimax-optimal excess-risk and strong RKHS convergence rates for general losses, avoiding the saturation seen in standard kernel SGD, and demonstrate favorable time and memory complexity when the minimizer is smooth. Numerical experiments on circular and higher-dimensional spherical data, as well as MNIST-derived non-spherical data, validate the theory and show tangible gains in both accuracy and computational efficiency.

Abstract

In this paper, we propose a novel kernel stochastic gradient descent (SGD) algorithm for large-scale supervised learning with general losses. Compared to traditional kernel SGD, our algorithm improves efficiency and scalability through an innovative regularization strategy. By leveraging the infinite series expansion of spherical radial basis functions, this strategy projects the stochastic gradient onto a finite-dimensional hypothesis space, which is adaptively scaled according to the bias-variance trade-off, thereby enhancing generalization performance. Based on a new estimation of the spectral structure of the kernel-induced covariance operator, we develop an analytical framework that unifies optimization and generalization analyses. We prove that both the last iterate and the suffix average converge at minimax-optimal rates, and we further establish optimal strong convergence in the reproducing kernel Hilbert space. Our framework accommodates a broad class of classical loss functions, including least-squares, Huber, and logistic losses. Moreover, the proposed algorithm significantly reduces computational complexity and achieves optimal storage complexity by incorporating coordinate-wise updates from linear SGD, thereby avoiding the costly pairwise operations typical of kernel SGD and enabling efficient processing of streaming data. Finally, extensive numerical experiments demonstrate the efficiency of our approach.

Truncated Kernel Stochastic Gradient Descent with General Losses and Spherical Radial Basis Functions

TL;DR

This work addresses scalable kernel-based online learning with general losses on spherical data by introducing Truncated Kernel SGD (T-kernel SGD). The method leverages spherical radial basis functions to decompose the RKHS into nested finite-dimensional spaces and projects stochastic gradients onto these spaces, enabling adaptive regularization and efficient updates. The authors prove minimax-optimal excess-risk and strong RKHS convergence rates for general losses, avoiding the saturation seen in standard kernel SGD, and demonstrate favorable time and memory complexity when the minimizer is smooth. Numerical experiments on circular and higher-dimensional spherical data, as well as MNIST-derived non-spherical data, validate the theory and show tangible gains in both accuracy and computational efficiency.

Abstract

In this paper, we propose a novel kernel stochastic gradient descent (SGD) algorithm for large-scale supervised learning with general losses. Compared to traditional kernel SGD, our algorithm improves efficiency and scalability through an innovative regularization strategy. By leveraging the infinite series expansion of spherical radial basis functions, this strategy projects the stochastic gradient onto a finite-dimensional hypothesis space, which is adaptively scaled according to the bias-variance trade-off, thereby enhancing generalization performance. Based on a new estimation of the spectral structure of the kernel-induced covariance operator, we develop an analytical framework that unifies optimization and generalization analyses. We prove that both the last iterate and the suffix average converge at minimax-optimal rates, and we further establish optimal strong convergence in the reproducing kernel Hilbert space. Our framework accommodates a broad class of classical loss functions, including least-squares, Huber, and logistic losses. Moreover, the proposed algorithm significantly reduces computational complexity and achieves optimal storage complexity by incorporating coordinate-wise updates from linear SGD, thereby avoiding the costly pairwise operations typical of kernel SGD and enabling efficient processing of streaming data. Finally, extensive numerical experiments demonstrate the efficiency of our approach.

Paper Structure

This paper contains 26 sections, 23 theorems, 179 equations, 4 figures, 1 table, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Works and Discussion
  3. Algorithm Overview and Main Contributions
  4. Preliminaries and Algorithm
  5. Loss Functions
  6. Spherical Radial Basis Functions
  7. Truncated Kernel Stochastic Gradient Descent
  8. Theoretical Results
  9. Optimal Rates for Excess Risk
  10. Optimal Rates for Strong Convergence
  11. Numerical Experiments
  12. Robust Regression on the Circle
  13. Robust Regression on 3-Dimensional Spherical Data
  14. Binary Classification of High-Dimensional Non-Spherical Data
  15. Appendix
  16. ...and 11 more sections

Key Result

Theorem 1

Assume that assum:extence (with $M>0$), assum:Lsmooth (with $L>0$), assum:mustorngconvex (with $\mu>0$), assum:independnt, assum:regularity condition (with $r\geq\frac{1}{2}$), and assum:Radon-Nikodym derivative (with $0<b_\rho<B_\rho$ in eq:equivalence norm) hold. Let $\theta=\frac{1}{2s(2r+1)}$ an where $\hat{f}_n$ denotes the last iterate in eq:generalized T-kernel SGD and $\bar{f}_{\alpha n} =

Figures (4)

  • Figure 1: The left figure illustrates the convergence of the error with respect to the sample size under three different losses, while the right figure shows the convergence of the error with respect to runtime. The black line indicates the minimax rate, with the slope $-\frac{7}{9}$.
  • Figure 2: The left figure illustrates the convergence of the error with respect to the sample size under three different losses, while the right figure shows the convergence of the error with respect to runtime. The black line indicates the minimax rate, with the slope $-\frac{3}{5}$.
  • Figure 3: The left figure illustrates the convergence of the error with respect to the sample size under three different losses, while the right figure shows the convergence of the error with respect to runtime. The black line indicates the minimax rate, with the slope $-\frac{2}{3}$.
  • Figure 4: The two plots above show the sample-accuracy and time-accuracy, respectively.

Theorems & Definitions (43)

  • Theorem 1
  • Proposition 1
  • Definition 1
  • Lemma 1
  • Theorem 2
  • Proposition A.1
  • proof
  • Proposition A.2
  • proof
  • Lemma A.1
  • ...and 33 more