Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Low-Rank Extragradient Method for Nonsmooth and Low-Rank Matrix Optimization Problems

Dan Garber, Atara Kaplan

TL;DR

This work tackles nonsmooth low-rank matrix optimization over the spectrahedron by recasting it as a smooth convex-concave saddle-point problem under generalized strict complementarity. The authors prove that the projected extragradient method can achieve the canonical $O(1/t)$ convergence rate while using only two low-rank SVDs per iteration, provided a warm-start and a suitable rank-$r$ projection. A precise radius $R_0(r)$ governs initialization to ensure rank preservation, and practical rank certificates enable inexpensive verification of low-rank projections. Empirical results across Sparse PCA, robust/low-rank matrix recovery, phase synchronization, and related tasks show that rank-$r$ truncated projections can replicate full-rank iterates and enable scalable optimization without sacrificing accuracy. The approach thus offers a principled and efficient pathway for nonsmooth, large-scale low-rank matrix optimization with strong theoretical and empirical support.

Abstract

Low-rank and nonsmooth matrix optimization problems capture many fundamental tasks in statistics and machine learning. While significant progress has been made in recent years in developing efficient methods for \textit{smooth} low-rank optimization problems that avoid maintaining high-rank matrices and computing expensive high-rank SVDs, advances for nonsmooth problems have been slow paced. In this paper we consider standard convex relaxations for such problems. Mainly, we prove that under a natural \textit{generalized strict complementarity} condition and under the relatively mild assumption that the nonsmooth objective can be written as a maximum of smooth functions, the \textit{extragradient method}, when initialized with a ``warm-start'' point, converges to an optimal solution with rate $O(1/t)$ while requiring only two \textit{low-rank} SVDs per iteration. We give a precise trade-off between the rank of the SVDs required and the radius of the ball in which we need to initialize the method. We support our theoretical results with empirical experiments on several nonsmooth low-rank matrix recovery tasks, demonstrating that using simple initializations, the extragradient method produces exactly the same iterates when full-rank SVDs are replaced with SVDs of rank that matches the rank of the (low-rank) ground-truth matrix to be recovered.

Low-Rank Extragradient Method for Nonsmooth and Low-Rank Matrix Optimization Problems

TL;DR

This work tackles nonsmooth low-rank matrix optimization over the spectrahedron by recasting it as a smooth convex-concave saddle-point problem under generalized strict complementarity. The authors prove that the projected extragradient method can achieve the canonical convergence rate while using only two low-rank SVDs per iteration, provided a warm-start and a suitable rank- projection. A precise radius governs initialization to ensure rank preservation, and practical rank certificates enable inexpensive verification of low-rank projections. Empirical results across Sparse PCA, robust/low-rank matrix recovery, phase synchronization, and related tasks show that rank- truncated projections can replicate full-rank iterates and enable scalable optimization without sacrificing accuracy. The approach thus offers a principled and efficient pathway for nonsmooth, large-scale low-rank matrix optimization with strong theoretical and empirical support.

Abstract

Low-rank and nonsmooth matrix optimization problems capture many fundamental tasks in statistics and machine learning. While significant progress has been made in recent years in developing efficient methods for \textit{smooth} low-rank optimization problems that avoid maintaining high-rank matrices and computing expensive high-rank SVDs, advances for nonsmooth problems have been slow paced. In this paper we consider standard convex relaxations for such problems. Mainly, we prove that under a natural \textit{generalized strict complementarity} condition and under the relatively mild assumption that the nonsmooth objective can be written as a maximum of smooth functions, the \textit{extragradient method}, when initialized with a ``warm-start'' point, converges to an optimal solution with rate while requiring only two \textit{low-rank} SVDs per iteration. We give a precise trade-off between the rank of the SVDs required and the radius of the ball in which we need to initialize the method. We support our theoretical results with empirical experiments on several nonsmooth low-rank matrix recovery tasks, demonstrating that using simple initializations, the extragradient method produces exactly the same iterates when full-rank SVDs are replaced with SVDs of rank that matches the rank of the (low-rank) ground-truth matrix to be recovered.
Paper Structure (19 sections, 14 theorems, 88 equations, 5 tables, 1 algorithm)

This paper contains 19 sections, 14 theorems, 88 equations, 5 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Additional related work
  3. Strict Complementarity for Nonsmooth Optimization and Difficulty of Applying Low-Rank Projected Subgradient Steps
  4. The challenge of applying low-rank projected subgradient steps
  5. From Nonsmooth to Saddle-Point Problems
  6. Projected Extragradient Method with Low-Rank Projections
  7. Back to nonsmooth problems
  8. Efficiently-computable certificates for correctness of low-rank projections
  9. Empirical Evidence
  10. Sparse PCA
  11. Low-rank and sparse matrix recovery
  12. Robust PCA
  13. Phase synchronization
  14. Linearly constrained low-rank matrix estimation
  15. Discussion
  16. ...and 4 more sections

Key Result

Lemma 1

Let $g:\mathbb{S}^n\rightarrow\mathbb{R}$ be a convex function. Then ${\mathbf{X}}^*\in{\mathcal{S}_n}$ minimizes $g$ over ${\mathcal{S}_n}$ if and only if there exists a subgradient ${\mathbf{G}}^*\in\partial g({\mathbf{X}}^*)$ such that $\langle {\mathbf{X}}-{\mathbf{X}}^*,{\mathbf{G}}^*\rangle\ge

Theorems & Definitions (26)

  • Lemma 1: first-order optimality condition, see beckOptimizationBook
  • Definition 1: strict complementarity
  • Lemma 2
  • Lemma 3
  • Lemma 4
  • Lemma 5: failure of low-rank subgradient descent on sparse PCA
  • proof
  • Lemma 6
  • Remark 1
  • Lemma 7
  • ...and 16 more