Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

A Rank-One-Update Method for the Training of Support Vector Machines

Florian Jarre

TL;DR

The paper addresses high-accuracy training of SVMs by solving a structured convex quadratic program and introduces CMU, a cycling active-set method that alternates between down-cycles (Newton steps with rank-one Cholesky updates) and up-cycles (controlled increases of the inactive set). The approach exploits the problem's structure to achieve descent with inexpensive updates, and it demonstrates global convergence and practical efficiency, often requiring only a handful of cycles. Numerical experiments on Gaussian-kernel SVMs with challenging ill-conditioning show CMU can outperform greedy and random SMO variants in accuracy and speed under moderate data sizes (up to about 10^4 points), while highlighting the role of regularization and iterative refinement for numerical stability. The work provides a compelling alternative to SMO when high-accuracy solutions are desirable and data sizes are moderate, with clear guidance on algorithmic choices and practical considerations for stability.

Abstract

This paper considers convex quadratic programs associated with the training of support vector machines (SVM). Exploiting the special structure of the SVM problem a new type of active set method with long cycles and stable rank-one-updates is proposed and tested (CMU: cycling method with updates). The structure of the problem allows for a repeated simple increase of the set of inactive constraints while controlling its size. This is followed by minimization steps with cheap updates of a matrix factorization. A widely used approach for solving SVM problems is the alternating direction method SMO, a method that is very efficient for low accuracy. The new active set approach allows for higher accuracy results at moderate computational cost. To relate both approaches, the effect of the accuracy on the running time and on the predictive quality of the SVM is compared with some numerical examples. A surprising result of the numerical examples is that only a very small number of cycles (each consisting of less than 2n steps) was used for CMU.

A Rank-One-Update Method for the Training of Support Vector Machines

TL;DR

The paper addresses high-accuracy training of SVMs by solving a structured convex quadratic program and introduces CMU, a cycling active-set method that alternates between down-cycles (Newton steps with rank-one Cholesky updates) and up-cycles (controlled increases of the inactive set). The approach exploits the problem's structure to achieve descent with inexpensive updates, and it demonstrates global convergence and practical efficiency, often requiring only a handful of cycles. Numerical experiments on Gaussian-kernel SVMs with challenging ill-conditioning show CMU can outperform greedy and random SMO variants in accuracy and speed under moderate data sizes (up to about 10^4 points), while highlighting the role of regularization and iterative refinement for numerical stability. The work provides a compelling alternative to SMO when high-accuracy solutions are desirable and data sizes are moderate, with clear guidance on algorithmic choices and practical considerations for stability.

Abstract

This paper considers convex quadratic programs associated with the training of support vector machines (SVM). Exploiting the special structure of the SVM problem a new type of active set method with long cycles and stable rank-one-updates is proposed and tested (CMU: cycling method with updates). The structure of the problem allows for a repeated simple increase of the set of inactive constraints while controlling its size. This is followed by minimization steps with cheap updates of a matrix factorization. A widely used approach for solving SVM problems is the alternating direction method SMO, a method that is very efficient for low accuracy. The new active set approach allows for higher accuracy results at moderate computational cost. To relate both approaches, the effect of the accuracy on the running time and on the predictive quality of the SVM is compared with some numerical examples. A surprising result of the numerical examples is that only a very small number of cycles (each consisting of less than 2n steps) was used for CMU.

Paper Structure

This paper contains 23 sections, 2 theorems, 35 equations.

Table of Contents

  1. Introduction
  2. Notation
  3. Basic problem in the training of support vector machines
  4. Optimality conditions
  5. An active set rank-one-update algorithm
  6. Outline
  7. Up-cycle, leaving active constraints
  8. Computing a search direction:
  9. Line search
  10. Summary, up-cycle
  11. Starting point
  12. Down-cycle, Newton iterations
  13. Cholesky updates
  14. Global convergence
  15. Technical details
  16. ...and 8 more sections

Key Result

Proposition 1

Let $x\in{\cal{B}}\cap{\cal{N}}$, $g=Hx-c$, and Set $\widetilde{g}:=g-\mu z$. Then $x$ is an optimal solution of (qp) if, and only if, $\sigma\circ\widetilde{g}\ge 0$ and $\widetilde{g}_{\mathbf{K}}=0$. (Here, $0\in{\mathbb{R}}^{|{\mathbf{K}}|}$ and for ${\mathbf{K}}=\emptyset$ the condition $\widetilde{g}_{\mathbf{K}}=0$ is trivially satisfied

Theorems & Definitions (6)

  • Proposition 1
  • Proposition 2
  • Remark 1
  • Remark 2
  • Remark 3
  • Remark 4