Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Classification by sparse generalized additive models

Felix Abramovich

TL;DR

The paper develops sparse generalized additive model (SpAM) classifiers for binary outcomes by minimizing the logistic loss with sparse group Lasso/Slope penalties on univariate additive components expanded in orthonormal bases. The proposed approach is adaptive to unknown sparsity and smoothness, and the authors prove nearly-minimax misclassification risk across analytic, Sobolev, and Besov function classes under a sparse group restricted eigenvalue condition, with a phase transition between sparse and dense regimes. They provide upper and lower bounds that match up to log factors and validate the theory with simulations and a real-data example (email spam). The work advances nonparametric, scalable classification in high dimensions by enabling simultaneous feature selection and smooth function estimation within a convex optimization framework.

Abstract

We consider (nonparametric) sparse (generalized) additive models (SpAM) for classification. The design of a SpAM classifier is based on minimizing the logistic loss with a sparse group Lasso/Slope-type penalties on the coefficients of univariate additive components' expansions in orthonormal series (e.g., Fourier or wavelets). The resulting classifier is inherently adaptive to the unknown sparsity and smoothness. We show that under certain sparse group restricted eigenvalue condition it is nearly-minimax (up to log-factors) simultaneously across the entire range of analytic, Sobolev and Besov classes. The performance of the proposed classifier is illustrated on a simulated and a real-data examples.

Classification by sparse generalized additive models

TL;DR

The paper develops sparse generalized additive model (SpAM) classifiers for binary outcomes by minimizing the logistic loss with sparse group Lasso/Slope penalties on univariate additive components expanded in orthonormal bases. The proposed approach is adaptive to unknown sparsity and smoothness, and the authors prove nearly-minimax misclassification risk across analytic, Sobolev, and Besov function classes under a sparse group restricted eigenvalue condition, with a phase transition between sparse and dense regimes. They provide upper and lower bounds that match up to log factors and validate the theory with simulations and a real-data example (email spam). The work advances nonparametric, scalable classification in high dimensions by enabling simultaneous feature selection and smooth function estimation within a convex optimization framework.

Abstract

We consider (nonparametric) sparse (generalized) additive models (SpAM) for classification. The design of a SpAM classifier is based on minimizing the logistic loss with a sparse group Lasso/Slope-type penalties on the coefficients of univariate additive components' expansions in orthonormal series (e.g., Fourier or wavelets). The resulting classifier is inherently adaptive to the unknown sparsity and smoothness. We show that under certain sparse group restricted eigenvalue condition it is nearly-minimax (up to log-factors) simultaneously across the entire range of analytic, Sobolev and Besov classes. The performance of the proposed classifier is illustrated on a simulated and a real-data examples.
Paper Structure (15 sections, 6 theorems, 55 equations, 2 tables)

This paper contains 15 sections, 6 theorems, 55 equations, 2 tables.

Table of Contents

  1. Introduction
  2. SpAM classifier
  3. Minimaxity across various function classes
  4. Sobolev classes
  5. Analytic functions
  6. Besov classes
  7. Examples
  8. Simulations
  9. Real-data example
  10. Appendix
  11. Proofs of the upper bounds
  12. Proof of Theorem \ref{['th:sobolevupper']}
  13. Proof of Theorem \ref{['th:analyticupper']}
  14. Proof of Theorem \ref{['th:besovupper']}
  15. Proofs of the lower bounds (Theorems \ref{['th:sobolevlower']}, \ref{['th:analyticlower']} and \ref{['th:besovlower']})

Key Result

Theorem 3.2

Consider the SpAM (eq:logistic)-(eq:SpAM), where $g \in {\cal G}_{\widetilde{H}}(d_0,{\bf s})$, and assume $WRE(d_0,\bm,c_0)$-condition as:design with $m_j=n^{\frac{1}{2s_j+1}},\;j=1,\ldots,d_0$ and $c_0$ that can be derived from the proof.

Theorems & Definitions (6)

  • Theorem 3.2
  • Theorem 3.3
  • Theorem 3.4
  • Theorem 3.5
  • Theorem 3.6
  • Theorem 3.7