Local Duality for Sparse Support Vector Machines
Penghe Zhang, Naihua Xiu, Houduo Qi
TL;DR
The paper addresses the theoretical gap for sparsity-driven dual SVMs by developing a local duality framework that links the $\ell_0$-regularized sparse SVM (SSVM) to the dual of the $0/1$-loss SVM. It introduces the concept of locally nice primal–dual pairs, proving that the dual of the $0/1$-loss SVM corresponds to an $\ell_0$-regularized SSVM and establishing a local linear representer for the primal solution. It further elucidates the connections among $0/1$-loss SVM, hinge-loss SVM with positive weights, and ramp-loss SVM, showing how local optima in one formulation map to the others under appropriate parameter choices and convergence conditions. Numerical experiments on real datasets validate that locally nice solutions can outperform standard convex SVMs and ramp-loss variants, offering practical guidance for hyperparameter selection. Overall, the work provides a theoretical foundation for using sparsity in the dual and offers explanation for observed empirical advantages of SSVMs.
Abstract
Due to the rise of cardinality minimization in optimization, sparse support vector machines (SSVMs) have attracted much attention lately and show certain empirical advantages over convex SVMs. A common way to derive an SSVM is to add a cardinality function such as $\ell_0$-norm to the dual problem of a convex SVM. However, this process lacks theoretical justification. This paper fills the gap by developing a local duality theory for such an SSVM formulation and exploring its relationship with the hinge-loss SVM (hSVM) and the ramp-loss SVM (rSVM). In particular, we prove that the derived SSVM is exactly the dual problem of the 0/1-loss SVM, and the linear representer theorem holds for their local solutions. The local solution of SSVM also provides guidelines on selecting hyperparameters of hSVM and rSVM. {Under specific conditions, we show that a sequence of global solutions of hSVM converges to a local solution of 0/1-loss SVM. Moreover, a local minimizer of 0/1-loss SVM is a local minimizer of rSVM.} This explains why a local solution induced by SSVM outperforms hSVM and rSVM in the prior empirical study. We further conduct numerical tests on real datasets and demonstrate potential advantages of SSVM by working with locally nice solutions proposed in this paper.