Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

A fast algorithm for solving the lasso problem exactly without homotopy using differential inclusions

Gabriel P. Langlois, Jérôme Darbon

TL;DR

This paper addresses solving the lasso problem exactly without homotopy by recasting the dual into a trajectory of a projected dynamical system through a minimal selection principle. It develops a slow-system differential inclusion that is integrable, yielding an exact algorithm (Algorithm 1) to compute primal and dual solutions for any $t\ge 0$ and to efficiently compute regularization paths. A continuation analysis in the hyperparameter $t$ provides a generalized homotopy algorithm, with finite-time convergence of the slow dynamics and explicit recovery of the primal solution from NNLS subproblems. Numerical experiments demonstrate superior accuracy and speed compared with state-of-the-art methods and suggest the framework extends to a broader class of polyhedral-constrained convex problems.

Abstract

We prove in this work that the well-known lasso problem can be solved exactly without homotopy using novel differential inclusions techniques. Specifically, we show that a selection principle from the theory of differential inclusions transforms the dual lasso problem into the problem of calculating the trajectory of a projected dynamical system that we prove is integrable. Our analysis yields an exact algorithm for the lasso problem, numerically up to machine precision, that is amenable to computing regularization paths and is very fast. Moreover, we show the continuation of solutions to the integrable projected dynamical system in terms of the hyperparameter naturally yields a rigorous homotopy algorithm. Numerical experiments confirm that our algorithm outperforms the state-of-the-art algorithms in both efficiency and accuracy. Beyond this work, we expect our results and analysis can be adapted to compute exact or approximate solutions to a broader class of polyhedral-constrained optimization problems.

A fast algorithm for solving the lasso problem exactly without homotopy using differential inclusions

TL;DR

This paper addresses solving the lasso problem exactly without homotopy by recasting the dual into a trajectory of a projected dynamical system through a minimal selection principle. It develops a slow-system differential inclusion that is integrable, yielding an exact algorithm (Algorithm 1) to compute primal and dual solutions for any and to efficiently compute regularization paths. A continuation analysis in the hyperparameter provides a generalized homotopy algorithm, with finite-time convergence of the slow dynamics and explicit recovery of the primal solution from NNLS subproblems. Numerical experiments demonstrate superior accuracy and speed compared with state-of-the-art methods and suggest the framework extends to a broader class of polyhedral-constrained convex problems.

Abstract

We prove in this work that the well-known lasso problem can be solved exactly without homotopy using novel differential inclusions techniques. Specifically, we show that a selection principle from the theory of differential inclusions transforms the dual lasso problem into the problem of calculating the trajectory of a projected dynamical system that we prove is integrable. Our analysis yields an exact algorithm for the lasso problem, numerically up to machine precision, that is amenable to computing regularization paths and is very fast. Moreover, we show the continuation of solutions to the integrable projected dynamical system in terms of the hyperparameter naturally yields a rigorous homotopy algorithm. Numerical experiments confirm that our algorithm outperforms the state-of-the-art algorithms in both efficiency and accuracy. Beyond this work, we expect our results and analysis can be adapted to compute exact or approximate solutions to a broader class of polyhedral-constrained optimization problems.

Paper Structure

This paper contains 33 sections, 14 theorems, 111 equations, 2 figures, 2 tables, 2 algorithms.

Table of Contents

  1. Introduction
  2. Contributions of this paper:
  3. Related work:
  4. Organization of this paper:
  5. Setup
  6. Existence of solutions and optimality conditions to the lasso problem
  7. Structure of the optimality conditions
  8. Characterization via differential inclusions
  9. The minimal selection principle
  10. The minimal selection principle for the lasso problem
  11. Computing the minimal selection
  12. Dynamics of the slow system
  13. The minimal selection is a descent direction
  14. Explicit local solution of the slow system
  15. Continuation of the local solution
  16. ...and 18 more sections

Key Result

Theorem 3.1

Let $g\colon \mathbb{R}^m \to \mathbb{R}\cup\{+\infty\}$ be a proper, lower semicontinuous, and convex function and let $\boldsymbol{p}_{0} \in \mathrm{dom~} \partial g$. Consider the gradient inclusions Then, there exists a unique solution $\boldsymbol{p} \colon [0,+\infty) \mapsto \mathrm{dom~} \partial g$ satisfying eq:general_diff_inclusion. Moreover:

Figures (2)

  • Figure 1: Relative error of the dual objective function with respect to Algorithm 1 (first row), relative error in $\ell_{\infty}$-norm of the dual solution with respect to Algorithm 1 (second row), and number of nonzero components of the primal solutions at the end of the solution paths. The dataset used is # 548 from lorenz2015solving. Left: Default tolerances ($\text{thresh} = 10^{-4}$ for glmnet, $\text{RelTol} = 10^{-4}$ for mlasso, and a relative tolerance of $10^{-4}$ for fista). Right: Harsher tolerances ($\text{thresh} = 10^{-13}$ for glmnet, $\text{RelTol} = 10^{-8}$ for mlasso, and a relative tolerance of $10^{-8}$ for fista).
  • Figure 2: Relative error of the dual objective function with respect to Algorithm 1 (first row), relative error in $\ell_{\infty}$-norm of the dual solution with respect to Algorithm 1 (second row), and number of nonzero components of the primal solutions at the end of the solution path. The dataset used is # 474 from lorenz2015solving. Left: Default tolerances ($\text{thresh} = 10^{-4}$ for glmnet, $\text{RelTol} = 10^{-4}$ for mlasso, and a relative tolerance of $10^{-4}$ for fista). Right: Harsher tolerances ($\text{thresh} = 10^{-13}$ for glmnet, $\text{RelTol} = 10^{-8}$ for mlasso, and a relative tolerance of $10^{-8}$ for fista).

Theorems & Definitions (52)

  • Theorem 3.1: Existence and uniqueness of solutions to gradient inclusions
  • proof
  • Remark 3.1
  • Lemma 3.1
  • proof
  • Remark 3.2
  • Proposition 4.1
  • proof
  • Theorem 4.1
  • proof
  • ...and 42 more