Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

First-Order Sparse Convex Optimization: Better Rates with Sparse Updates

Dan Garber

Abstract

It was recently established that for convex optimization problems with sparse optimal solutions (be it entry-wise sparsity or matrix rank-wise sparsity) it is possible to design first-order methods with linear convergence rates that depend on an improved mixed-norm condition number of the form $\frac{β_1{}s}{α_2}$, where $β_1$ is the $\ell_1$-Lipschitz continuity constant of the gradient, $α_2$ is the $\ell_2$-quadratic growth constant, and $s$ is the sparsity of optimal solutions. However, beyond the improved convergence rate, these methods are unable to leverage the sparsity of optimal solutions towards improving the runtime of each iteration as well, which may still be prohibitively high for high-dimensional problems. In this work, we establish that linear convergence rates which depend on this improved condition number can be obtained using only sparse updates, which may result in overall significantly improved running times. Moreover, our methods are considerably easier to implement.

First-Order Sparse Convex Optimization: Better Rates with Sparse Updates

Abstract

It was recently established that for convex optimization problems with sparse optimal solutions (be it entry-wise sparsity or matrix rank-wise sparsity) it is possible to design first-order methods with linear convergence rates that depend on an improved mixed-norm condition number of the form , where is the -Lipschitz continuity constant of the gradient, is the -quadratic growth constant, and is the sparsity of optimal solutions. However, beyond the improved convergence rate, these methods are unable to leverage the sparsity of optimal solutions towards improving the runtime of each iteration as well, which may still be prohibitively high for high-dimensional problems. In this work, we establish that linear convergence rates which depend on this improved condition number can be obtained using only sparse updates, which may result in overall significantly improved running times. Moreover, our methods are considerably easier to implement.

Paper Structure

This paper contains 12 sections, 10 theorems, 36 equations, 1 figure, 1 table, 3 algorithms.

Table of Contents

  1. Introduction
  2. Unified Setup, Algorithm, and Convergence Analysis
  3. Applications
  4. $\ell_1$-ball setup
  5. Nuclear norm ball setup
  6. Numerical Evidence
  7. Open Questions
  8. Away-Steps Frank-Wolfe for Problem \ref{['eq:L1ProbInt']}
  9. Accelerated Rates with Dense Updates --- Recovering the rates in ilandarideva2024accelerated
  10. Proof of Lemma \ref{['lem:L1norms']}
  11. Proof of Lemma \ref{['lem:NNnorms']}
  12. Sparse Projections onto the Nuclear Norm Ball

Key Result

Theorem 1

Consider the iterates $({\mathbf{x}}_t)_{t\geq 1}$ of Algorithm alg:weakProxFW with a fixed step-size $\eta =\frac{\alpha_2}{4\beta(8s+s^*_{\mathcal{K}})}\in[0,1]$ and when $\gamma_t$ is set either via line-search or to the pre-defined value $\gamma_t = \eta$ for all $t$. Then,

Figures (1)

  • Figure 1: Empirical convergence of various algorithms for Problem \ref{['eq:L1ProbInt']}.

Theorems & Definitions (20)

  • Theorem 1
  • Remark 1
  • proof : Proof of Theorem \ref{['thm:conv']}
  • Lemma 1
  • Corollary 1
  • Remark 2
  • Lemma 2
  • Corollary 2
  • Remark 3
  • Theorem 2: Away-Steps Frank-Wolfe for Problem \ref{['eq:L1ProbInt']}
  • ...and 10 more