Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

The Stochastic Conjugate Subgradient Algorithm For Kernel Support Vector Machines

Di Zhang, Suvrajeet Sen

TL;DR

This work tackles large‑scale kernel SVM optimization where standard stochastic first‑order methods may struggle with non‑smooth, finite‑sum objectives. It introduces the Stochastic Conjugate Subgradient (SCS) algorithm, which fuses Wolfe's non‑smooth conjugate subgradient ideas with adaptive sequential sampling and a decomposition of the objective to efficiently solve a sequence of local approximations $f_k$ using expanding data subsets. The authors derive a rigorous convergence theory, proving a rate of $O(1/\varepsilon^2)$ and establishing an optimality condition tied to the stopping criterion $\|d_k\|<\varepsilon$, supported by a supermartingale framework and renewal arguments. Empirical results on UCI datasets show that SCS achieves lower objective values and better scalability than conventional SFO methods, particularly on very large datasets, indicating meaningful practical impact for large‑scale kernel SVMs and potentially other non‑smooth stochastic convex problems.

Abstract

Stochastic First-Order (SFO) methods have been a cornerstone in addressing a broad spectrum of modern machine learning (ML) challenges. However, their efficacy is increasingly questioned, especially in large-scale applications where empirical evidence indicates potential performance limitations. In response, this paper proposes an innovative method specifically designed for kernel support vector machines (SVMs). This method not only achieves faster convergence per iteration but also exhibits enhanced scalability when compared to conventional SFO techniques. Diverging from traditional sample average approximation strategies that typically frame kernel SVM as an 'all-in-one' Quadratic Program (QP), our approach adopts adaptive sampling. This strategy incrementally refines approximation accuracy on an 'as-needed' basis. Crucially, this approach also inspires a decomposition-based algorithm, effectively decomposing parameter selection from error estimation, with the latter being independently determined for each data point. To exploit the quadratic nature of the kernel matrix, we introduce a stochastic conjugate subgradient method. This method preserves many benefits of first-order approaches while adeptly handling both nonlinearity and non-smooth aspects of the SVM problem. Thus, it extends beyond the capabilities of standard SFO algorithms for non-smooth convex optimization. The convergence rate of this novel method is thoroughly analyzed within this paper. Our experimental results demonstrate that the proposed algorithm not only maintains but potentially exceeds the scalability of SFO methods. Moreover, it significantly enhances both speed and accuracy of the optimization process.

The Stochastic Conjugate Subgradient Algorithm For Kernel Support Vector Machines

TL;DR

This work tackles large‑scale kernel SVM optimization where standard stochastic first‑order methods may struggle with non‑smooth, finite‑sum objectives. It introduces the Stochastic Conjugate Subgradient (SCS) algorithm, which fuses Wolfe's non‑smooth conjugate subgradient ideas with adaptive sequential sampling and a decomposition of the objective to efficiently solve a sequence of local approximations using expanding data subsets. The authors derive a rigorous convergence theory, proving a rate of and establishing an optimality condition tied to the stopping criterion , supported by a supermartingale framework and renewal arguments. Empirical results on UCI datasets show that SCS achieves lower objective values and better scalability than conventional SFO methods, particularly on very large datasets, indicating meaningful practical impact for large‑scale kernel SVMs and potentially other non‑smooth stochastic convex problems.

Abstract

Stochastic First-Order (SFO) methods have been a cornerstone in addressing a broad spectrum of modern machine learning (ML) challenges. However, their efficacy is increasingly questioned, especially in large-scale applications where empirical evidence indicates potential performance limitations. In response, this paper proposes an innovative method specifically designed for kernel support vector machines (SVMs). This method not only achieves faster convergence per iteration but also exhibits enhanced scalability when compared to conventional SFO techniques. Diverging from traditional sample average approximation strategies that typically frame kernel SVM as an 'all-in-one' Quadratic Program (QP), our approach adopts adaptive sampling. This strategy incrementally refines approximation accuracy on an 'as-needed' basis. Crucially, this approach also inspires a decomposition-based algorithm, effectively decomposing parameter selection from error estimation, with the latter being independently determined for each data point. To exploit the quadratic nature of the kernel matrix, we introduce a stochastic conjugate subgradient method. This method preserves many benefits of first-order approaches while adeptly handling both nonlinearity and non-smooth aspects of the SVM problem. Thus, it extends beyond the capabilities of standard SFO algorithms for non-smooth convex optimization. The convergence rate of this novel method is thoroughly analyzed within this paper. Our experimental results demonstrate that the proposed algorithm not only maintains but potentially exceeds the scalability of SFO methods. Moreover, it significantly enhances both speed and accuracy of the optimization process.
Paper Structure (15 sections, 13 theorems, 75 equations, 2 figures, 4 tables, 2 algorithms)

This paper contains 15 sections, 13 theorems, 75 equations, 2 figures, 4 tables, 2 algorithms.

Table of Contents

  1. Introduction
  2. Stochastic first-order methods
  3. Stochastic methods for general non-linear optimization
  4. Stochastic conjugate gradient method
  5. Contributions
  6. Stochastic Conjugate Subgradient (SCS) Algorithm
  7. Convergence Properties
  8. Stochastic Local Approximations
  9. Sufficient Decreases and Supermartingale Property
  10. Convergence Rate
  11. Optimality Condition
  12. Preliminary Computational Results
  13. Conclusion
  14. Proof of optimality condition
  15. Notation Tables

Key Result

Theorem 3.3

\newlabelL^00 Assume that $f$ and $f_k$ are Lipschitz continuous with constant $L_f$, for any $0 < \varepsilon <1$, $\kappa > \frac{4L_f}{\delta_{min}}$. If for all $i$, we have $|\langle \beta, \phi(z_i)\rangle| \leq M$ and then

Figures (2)

  • Figure 1: $\{f(\hat{\alpha}_k)\}_{k=1}^{k=50}$ for different combinations (data,algorithm).
  • Figure 2: $\{f(\hat{\alpha}_k)\}_{k=-50}^{k=-1}$ for different combinations (data,algorithm).

Theorems & Definitions (30)

  • Definition 3.1
  • Definition 3.2
  • Theorem 3.3
  • Proof 1
  • Definition 3.4
  • Definition 3.5
  • Lemma 3.6
  • Proof 2
  • Lemma 3.7
  • Proof 3
  • ...and 20 more