Binary Kernel Logistic Regression: a sparsity-inducing formulation and a convergent decomposition training algorithm

TL;DR

This work tackles the sparsity challenge in Kernel Logistic Regression (KLR) by introducing a sparsity-inducing dual formulation that incorporates a primal slack variable $\rho$ and a sparsity penalty, yielding a bounded, convex dual with a direct link $\rho(\lambda)=\lambda$ to a dual hyperparameter. An SMO-type decomposition algorithm leveraging second-order information is developed to solve the dual efficiently, with global convergence guarantees under PSD kernels. Empirical results on 12 datasets show that the proposed Sparse KLR (S-KLR) achieves accuracy on par with SVM while using substantially fewer kernel terms, and outperforms Import Vector Machine (IVM) and $\ell_{1/2}$-KLR on sparsity-accuracy trade-offs; second-order WSS also reduces training time significantly. The approach preserves probabilistic class membership estimates and offers practical scalability through bounded duals and principled sparsity control, suggesting avenues for kernel caching, low-rank approximations, and multiclass extensions in future work.

Abstract

Kernel logistic regression (KLR) is a widely used supervised learning method for binary and multi-class classification, which provides estimates of the conditional probabilities of class membership for the data points. Unlike other kernel methods such as Support Vector Machines (SVMs), KLRs are generally not sparse. Previous attempts to deal with sparsity in KLR include a heuristic method referred to as the Import Vector Machine (IVM) and ad hoc regularizations such as the $\ell_{1/2}$-based one. Achieving a good trade-off between prediction accuracy and sparsity is still a challenging issue with a potential significant impact from the application point of view. In this work, we revisit binary KLR and propose an extension of the training formulation proposed by Keerthi et al., which is able to induce sparsity in the trained model, while maintaining good testing accuracy. To efficiently solve the dual of this formulation, we devise a decomposition algorithm of Sequential Minimal Optimization type which exploits second-order information, and for which we establish global convergence. Numerical experiments conducted on 12 datasets from the literature show that the proposed binary KLR approach achieves a competitive trade-off between accuracy and sparsity with respect to IVM, $\ell_{1/2}$-based regularization for KLR, and SVM while retaining the advantages of providing informative estimates of the class membership probabilities.

Figures (8)

• Figure 1: Plots of the Hinge loss used in Support Vector Machines in blue and of the Binomial Negative Log-Likelihood used in logistic regression in red.
• Figure 2: Illustration of the negative log-likelihood function and its second derivative with respect to the parameter $\xi$.
• Figure 3: Scatterplots of the Synth dataset with 2 classes (red and blue), $p=2$ and $N=99$, for four values of the hyperparameter $\nu \in \{0,500,562.5,625\}$. Setting $\nu=0$ of the top-left plot amounts to solving the original formulation \ref{['eq:k-primal']}.Data points selected by sparsity-inducing formulation (with nonzero $\alpha_i$) are represented with bold red rhombuses and bold blue crosses, discarded/excluded data points are represented with thin blue crosses and red small squares.
• Figure 4: Plots of the number of data points selected by the sparsity-inducing formulation (in blue) and of the norm of the vector $\bm{\omega}$ (in red) as functions of the hyperparameter $\nu$.
• Figure 5: Scatter plot of the Synth dataset. The three lines represent the decision boundary related to Sparse KLR (blue dashed line), Thresholding heuristic (orange solid line), and the KLR formulation with $\nu=0$ (green dashed line). The rhombuses and crosses represent the selected data points for the red and blue classes, respectively. The cyan data point is selected solely by the sparse formulation while the black one is selected by imposing the threshold.

Theorems & Definitions (11)

• Lemma 1
• Proposition 1
• Proposition 2
• Lemma 2
• Lemma 3
• Proposition 3
• Corollary 1
• Lemma 1
• Proposition 1
• Lemma 4