A Novel Loss Function-based Support Vector Machine for Binary Classification
Yan Li, Liping Zhang
TL;DR
The paper introduces the Slide loss $\ell_s$ to address insufficient penalization in classic SVM losses, enabling graded penalties for samples inside the margin and near the decision boundary. It develops the $\ell_s$-SVM, derives the subdifferential and proximal operator of $\ell_s$, and defines proximal stationary points to establish first-order optimality conditions. An efficient $\ell_s$-ADMM algorithm with a behavior-based working set is proposed and analyzed, including convergence to proximal stationary points and local minima. Extensive experiments on seven real-world datasets demonstrate improved robustness and generalization over multiple SVM solvers, especially under label noise.
Abstract
The previous support vector machine(SVM) including $0/1$ loss SVM, hinge loss SVM, ramp loss SVM, truncated pinball loss SVM, and others, overlooked the degree of penalty for the correctly classified samples within the margin. This oversight affects the generalization ability of the SVM classifier to some extent. To address this limitation, from the perspective of confidence margin, we propose a novel Slide loss function ($\ell_s$) to construct the support vector machine classifier($\ell_s$-SVM). By introducing the concept of proximal stationary point, and utilizing the property of Lipschitz continuity, we derive the first-order optimality conditions for $\ell_s$-SVM. Based on this, we define the $\ell_s$ support vectors and working set of $\ell_s$-SVM. To efficiently handle $\ell_s$-SVM, we devise a fast alternating direction method of multipliers with the working set ($\ell_s$-ADMM), and provide the convergence analysis. The numerical experiments on real world datasets confirm the robustness and effectiveness of the proposed method.