Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Exterior-point Optimization for Sparse and Low-rank Optimization

Shuvomoy Das Gupta, Bartolomeo Stellato, Bart P. G. Van Parys

Abstract

Many problems of substantial current interest in machine learning, statistics, and data science can be formulated as sparse and low-rank optimization problems. In this paper, we present the nonconvex exterior-point optimization solver NExOS -- a first-order algorithm tailored to sparse and low-rank optimization problems. We consider the problem of minimizing a convex function over a nonconvex constraint set, where the set can be decomposed as the intersection of a compact convex set and a nonconvex set involving sparse or low-rank constraints. Unlike the convex relaxation approaches, NExOS finds a locally optimal point of the original problem by solving a sequence of penalized problems with strictly decreasing penalty parameters by exploiting the nonconvex geometry. NExOS solves each penalized problem by applying a first-order algorithm, which converges linearly to a local minimum of the corresponding penalized formulation under regularity conditions. Furthermore, the local minima of the penalized problems converge to a local minimum of the original problem as the penalty parameter goes to zero. We then implement and test NExOS on many instances from a wide variety of sparse and low-rank optimization problems, empirically demonstrating that our algorithm outperforms specialized methods.

Exterior-point Optimization for Sparse and Low-rank Optimization

Abstract

Many problems of substantial current interest in machine learning, statistics, and data science can be formulated as sparse and low-rank optimization problems. In this paper, we present the nonconvex exterior-point optimization solver NExOS -- a first-order algorithm tailored to sparse and low-rank optimization problems. We consider the problem of minimizing a convex function over a nonconvex constraint set, where the set can be decomposed as the intersection of a compact convex set and a nonconvex set involving sparse or low-rank constraints. Unlike the convex relaxation approaches, NExOS finds a locally optimal point of the original problem by solving a sequence of penalized problems with strictly decreasing penalty parameters by exploiting the nonconvex geometry. NExOS solves each penalized problem by applying a first-order algorithm, which converges linearly to a local minimum of the corresponding penalized formulation under regularity conditions. Furthermore, the local minima of the penalized problems converge to a local minimum of the original problem as the penalty parameter goes to zero. We then implement and test NExOS on many instances from a wide variety of sparse and low-rank optimization problems, empirically demonstrating that our algorithm outperforms specialized methods.

Paper Structure

This paper contains 62 sections, 10 theorems, 38 equations, 6 figures, 2 algorithms.

Table of Contents

  1. Introduction
  2. Sparsity-constrained optimization
  3. Low-rank optimization
  4. Related work
  5. Convex relaxation approach
  6. First-order methods
  7. Discrete optimization approach
  8. Contributions
  9. Organization of the paper
  10. Our approach
  11. Properties of Moreau envelope for a nonconvex set.
  12. Algorithm description
  13. Algorithm subroutines
  14. Computing proximal operator of $f$
  15. Computing projection onto $\mathcal{X}$
  16. ...and 47 more sections

Key Result

Lemma 1

Consider the nonconvex compact constraint set $\mathcal{X}$ in problem (eq:original_problem-1). Denote $\kappa=1/(\beta\gamma+1)\in[0,1]$ and $\theta=\mu/(\gamma\kappa+\mu)\in[0,1]$. Then, for any $x\in\mathbf{E}$, and for any $\mu,\beta,\gamma>0$, we have $\mathbf{prox}_{\gamma\prescript{\mu}{}{\ma

Figures (6)

  • Figure 1: An illustration of how the penalized cost function in problem \ref{['eq:smoothed-opt']} compares against the original cost function in problem \ref{['eq:original_problem-1']} for different values of $\mu$. Note that the regularization parameter $\beta$ is kept fixed at its initial value $1$ throughout.
  • Figure 2: Sparse regression problem: comparison between NExOS (shown in blue), glmnet (shown in red), and Gurobi (shown in green). The first and second rows correspond to SNR 6 and SNR 1, respectively. For each SNR, the first column compares support recovery, the second column shows how close the objective value of the solution found by each algorithm gets to the optimal objective value (normalized as 1), and the third column shows the solution time (s) of each algorithm.
  • Figure 3: RMS error vs $k$ (cardinality) for the weather prediction problem.
  • Figure 4: Affine rank minimization problem: comparison between solutions found by NExOS and NCVX algorithm by Diamond2018.
  • Figure 5: Matrix completion problem: comparison between solutions found by NExOS and NCVX algorithm by Diamond2018.
  • ...and 1 more figures

Theorems & Definitions (19)

  • Lemma 1: Computing $\mathbf{prox}_{\gamma\prescript{\mu}{}{\mathcal{I}}}(x)$
  • proof
  • proof
  • proof
  • Lemma 2: Distance between local minima of problem (\ref{['eq:original_problem-1']}) with local minima of problem (\ref{['eq:smoothed-opt']})
  • proof
  • Theorem 1: Convergence result for NExOS
  • proof
  • Lemma 3: Convergence of the first inner algorithm
  • proof
  • ...and 9 more