Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Heaviside Low-Rank Support Matrix Machine

Xianchao Xiu, Shenghao Sun, Xinrong Li, Jiyuan Tao

TL;DR

A novel Heaviside low-rank SMM model called HL-SMM is proposed, which leverages the Heaviside loss instead of the common hinge or ramp losses for robustness and the low-rank constraint is adopted to accurately characterize the inherent global structure.

Abstract

Support matrix machine (SMM) is an emerging classification framework that directly handles matrix-structured observations, thereby avoiding the spatial correlations destroyed by vectorization. However, most existing SMM variants rely on convex or nonconvex surrogate loss functions, which may lead to high sensitivity to noise. To address this issue, we propose a novel Heaviside low-rank SMM model called HL-SMM, which leverages the Heaviside loss instead of the common hinge or ramp losses for robustness. Moreover, the low-rank constraint is adopted to accurately characterize the inherent global structure. In theory, we analyze the Karush-Kuhn-Tucker (KKT) points and rigorously prove the sufficient and necessary conditions. In algorithms, we develop an effective proximal alternating minimization (PAM) scheme, where all subproblems have closed-form solutions. Extensive experiments on benchmark datasets validate that the proposed HL-SMM achieves superior classification accuracy and robustness compared to state-of-the-art methods.

Heaviside Low-Rank Support Matrix Machine

TL;DR

A novel Heaviside low-rank SMM model called HL-SMM is proposed, which leverages the Heaviside loss instead of the common hinge or ramp losses for robustness and the low-rank constraint is adopted to accurately characterize the inherent global structure.

Abstract

Support matrix machine (SMM) is an emerging classification framework that directly handles matrix-structured observations, thereby avoiding the spatial correlations destroyed by vectorization. However, most existing SMM variants rely on convex or nonconvex surrogate loss functions, which may lead to high sensitivity to noise. To address this issue, we propose a novel Heaviside low-rank SMM model called HL-SMM, which leverages the Heaviside loss instead of the common hinge or ramp losses for robustness. Moreover, the low-rank constraint is adopted to accurately characterize the inherent global structure. In theory, we analyze the Karush-Kuhn-Tucker (KKT) points and rigorously prove the sufficient and necessary conditions. In algorithms, we develop an effective proximal alternating minimization (PAM) scheme, where all subproblems have closed-form solutions. Extensive experiments on benchmark datasets validate that the proposed HL-SMM achieves superior classification accuracy and robustness compared to state-of-the-art methods.
Paper Structure (14 sections, 6 theorems, 38 equations, 4 figures, 4 tables, 1 algorithm)

This paper contains 14 sections, 6 theorems, 38 equations, 4 figures, 4 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Preliminaries
  3. The Proposed Method
  4. New Formulation
  5. Optimality Conditions
  6. Optimization Algorithm
  7. Convergence Analysis
  8. Numerical Experiments
  9. Experimental Setup
  10. Experimental Results
  11. Discussions
  12. Parameter Analysis
  13. Convergence Visualization
  14. Conclusion

Key Result

Lemma 1

Given $\mathbf{W}\in\mathbb{R}^{p\times q}$ with singular values $\sigma_1(\mathbf{W})\ge\cdots\ge\sigma_s(\mathbf{W})>0$, the (possibly set-valued) projection onto the rank-constrained set is when $\sigma_r(\mathbf{W})>\sigma_{r+1}(\mathbf{W})$ (or $\sigma_r(\mathbf{W})=0$), the projection is single-valued.

Figures (4)

  • Figure 1: Coefficient matrices obtained by different loss functions.
  • Figure 2: Robustness comparison under 20% Gaussian noise.
  • Figure 3: Parameter sensitivity analysis of $r$ and $\beta$.
  • Figure 4: Convergence curves of the proposed algorithm.

Theorems & Definitions (11)

  • Lemma 1: Projection of $\mathcal{R}$
  • Lemma 2: Normal cone of $\mathcal{R}$
  • Lemma 3: Proximal operator of $\|(\cdot)_+\|_0$
  • Definition 1
  • Theorem 1: Necessary optimality condition
  • proof
  • Theorem 2: Suffcient optimality condition
  • proof
  • Theorem 3
  • proof
  • ...and 1 more