Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

A Fast Method for Lasso and Logistic Lasso

Siu-Wing Cheng, Man Ting Wong

TL;DR

This work introduces a fast, active-set based framework for solving Lasso, compressed sensing, and Logistic Lasso problems by iteratively solving reduced subproblems with most variables fixed at zero. The core idea is to update the active set using a principled KKT/subgradient criterion, freeing a small number of variables per iteration to maintain fast intermediate solves. The authors design a specific update rule with a tunable parameter $\tau$, backed by a heuristic dimensionality-reduction argument, and demonstrate large average speedups when hybridized with GPSR, ADMM, lassoglm, and glmnet across synthetic and real datasets. Empirically, the hybrids achieve substantial accelerations in compressed sensing (up to ~108x with ADMM and ~52x with glmnet), Lasso (tens of times), and Logistic Lasso (roughly 12x with lassoglm), highlighting practical impact for high-dimensional sparse recovery and classification tasks. The work also identifies avenues for improvement, including tighter solver integration and theoretical convergence analysis of the proposed method.

Abstract

We propose a fast method for solving compressed sensing, Lasso regression, and Logistic Lasso regression problems that iteratively runs an appropriate solver using an active set approach. We design a strategy to update the active set that achieves a large speedup over a single call of several solvers, including gradient projection for sparse reconstruction (GPSR), lassoglm of Matlab, and glmnet. For compressed sensing, the hybrid of our method and GPSR is 31.41 times faster than GPSR on average for Gaussian ensembles and 25.64 faster on average for binary ensembles. For Lasso regression, the hybrid of our method and GPSR achieves a 30.67-fold average speedup in our experiments. In our experiments on Logistic Lasso regression, the hybrid of our method and lassoglm gives an 11.95-fold average speedup, and the hybrid of our method and glmnet gives a 1.40-fold average speedup.

A Fast Method for Lasso and Logistic Lasso

TL;DR

This work introduces a fast, active-set based framework for solving Lasso, compressed sensing, and Logistic Lasso problems by iteratively solving reduced subproblems with most variables fixed at zero. The core idea is to update the active set using a principled KKT/subgradient criterion, freeing a small number of variables per iteration to maintain fast intermediate solves. The authors design a specific update rule with a tunable parameter , backed by a heuristic dimensionality-reduction argument, and demonstrate large average speedups when hybridized with GPSR, ADMM, lassoglm, and glmnet across synthetic and real datasets. Empirically, the hybrids achieve substantial accelerations in compressed sensing (up to ~108x with ADMM and ~52x with glmnet), Lasso (tens of times), and Logistic Lasso (roughly 12x with lassoglm), highlighting practical impact for high-dimensional sparse recovery and classification tasks. The work also identifies avenues for improvement, including tighter solver integration and theoretical convergence analysis of the proposed method.

Abstract

We propose a fast method for solving compressed sensing, Lasso regression, and Logistic Lasso regression problems that iteratively runs an appropriate solver using an active set approach. We design a strategy to update the active set that achieves a large speedup over a single call of several solvers, including gradient projection for sparse reconstruction (GPSR), lassoglm of Matlab, and glmnet. For compressed sensing, the hybrid of our method and GPSR is 31.41 times faster than GPSR on average for Gaussian ensembles and 25.64 faster on average for binary ensembles. For Lasso regression, the hybrid of our method and GPSR achieves a 30.67-fold average speedup in our experiments. In our experiments on Logistic Lasso regression, the hybrid of our method and lassoglm gives an 11.95-fold average speedup, and the hybrid of our method and glmnet gives a 1.40-fold average speedup.
Paper Structure (26 sections, 3 theorems, 4 equations, 2 figures, 8 tables, 1 algorithm)

This paper contains 26 sections, 3 theorems, 4 equations, 2 figures, 8 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Lasso Regression.
  3. Logistic Lasso Regression.
  4. Choice of solvers.
  5. Our contributions.
  6. Algorithm
  7. Notation.
  8. Duality and Slater's condition.
  9. Subgradient.
  10. KKT conditions.
  11. Algorithm.
  12. Why does a small $\tau$ work?
  13. Experimental results
  14. Compressed Sensing
  15. Gaussian ensemble matrix
  16. ...and 11 more sections

Key Result

Lemma 2.1

Let $\mathtt{z}$ be a feasible point of the primal problem. Let $I = \{i : (\mathtt{z})_i = 0\}$, let $I^+ = \{i : (\mathtt{z})_i > 0\}$, and let $I^- = \{i : (\mathtt{z})_i < 0\}$. Then, $\mathtt{g} \in \partial F(\mathtt{z})$ if and only if:

Figures (2)

  • Figure 1: The Lagrangian form of Lasso/Logistic Lasso and its dual problem.
  • Figure 2:

Theorems & Definitions (5)

  • Lemma 2.1
  • Lemma 2.2
  • proof
  • Lemma C.1
  • proof