$p$SVM: Soft-margin SVMs with $p$-norm Hinge Loss

TL;DR

The paper addresses binary and multiclass classification by proposing pSVM, a soft-margin SVM with a $p$-norm hinge loss ($p\ge1$), enabling flexible control of margin versus slack. It provides theoretical foundations including a generalization bound and a dual optimization problem with $\gamma=\tfrac{p}{p-1}$ and $\theta$, and introduces the pSMO family (notably $1.5$SMO and $2$SMO) to enable scalable training. Empirically, pSVM achieves superior accuracy across diverse binary and multiclass datasets, with best performance when $p$ lies roughly in $[1.25,2]$ and OvO multiclass implemented via pSVM classifiers. The work also supplies practical implementation guidance and publicly available code for reproducibility and extension.

Abstract

Support Vector Machines (SVMs) based on hinge loss have been extensively discussed and applied to various binary classification tasks. These SVMs achieve a balance between margin maximization and the minimization of slack due to outliers. Although many efforts have been dedicated to enhancing the performance of SVMs with hinge loss, studies on $p$SVMs, soft-margin SVMs with $p$-norm hinge loss, remain relatively scarce. In this paper, we explore the properties, performance, and training algorithms of $p$SVMs. We first derive the generalization bound of $p$SVMs, then formulate the dual optimization problem, comparing it with the traditional approach. Furthermore, we discuss a generalized version of the Sequential Minimal Optimization (SMO) algorithm, $p$SMO, to train our $p$SVM model. Comparative experiments on various datasets, including binary and multi-class classification tasks, demonstrate the effectiveness and advantages of our $p$SVM model and the $p$SMO method. Code is available at https://github.com/CoderBak/pSVM.

Figures (3)

• Figure 1: $\Phi_{p,\rho}$ under different $p$-values.
• Figure 2: Training process on the Wine Quality Dataset.
• Figure 3: Training process on USPS with $p=2,C=0.25$.