Fast low-rank estimation by projected gradient descent: General statistical and algorithmic guarantees

TL;DR

This paper provides a unified nonconvex framework for fast low-rank estimation via projected gradient descent on factorized matrix forms. By introducing natural conditions—M^*-faithfulness, local descent, local Lipschitz, and local smoothness—the authors prove sublinear and, under stronger smoothness, linear convergence to a statistically meaningful solution, without relying on local convexity. They instantiate the theory across a broad set of models, including matrix sensing, matrix completion (real and one-bit), sparse PCA, robust decomposition, and clustering, obtaining initialization strategies and sample-complexity bounds comparable to, and sometimes matching, convex relaxations. The results also demonstrate practical computational advantages, as updates scale with dr rather than d^2, and require no sample-splitting or repeated full SVDs. Collectively, the work advances understanding of when and how nonconvex factorized methods achieve optimal statistical accuracy with efficient computation.

Abstract

Optimization problems with rank constraints arise in many applications, including matrix regression, structured PCA, matrix completion and matrix decomposition problems. An attractive heuristic for solving such problems is to factorize the low-rank matrix, and to run projected gradient descent on the nonconvex factorized optimization problem. The goal of this problem is to provide a general theoretical framework for understanding when such methods work well, and to characterize the nature of the resulting fixed point. We provide a simple set of conditions under which projected gradient descent, when given a suitable initialization, converges geometrically to a statistically useful solution. Our results are applicable even when the initial solution is outside any region of local convexity, and even when the problem is globally concave. Working in a non-asymptotic framework, we show that our conditions are satisfied for a wide range of concrete models, including matrix regression, structured PCA, matrix completion with real and quantized observations, matrix decomposition, and graph clustering problems. Simulation results show excellent agreement with the theoretical predictions.

Figures (5)

• Figure 1: Simulation results for matrix completion. (a) Plots of optimization error $\textup{d} (F^{t}, F^{T} )$ and statistical error $\textup{d}(F^{t}, F^*)$ versus the iteration number $t$ using SVD initialization. Panel (b): same plots using a random initialization. The simulation is performed using $d = 1000$, $r=10$, $p=0.1$ and $\sigma = 0.01 \cdot \frac{r}{d}$. Panel (c): plots of per-entry estimation error $\frac{1}{d^2} \textup{d}(\widehat{F}, F^*)$ versus $\frac{r}{d}$, for different values of $(d, r)$ using SVD-based initialization. Each point represents the average over $20$ random instances. The simulation is performed using $p=0.1$ and $\sigma = 0.001$.
• Figure 2: Simulation results for sparse PCA. Panel (a); plots of optimization error $\textup{d} (F^{t}, F^{T} )$ and statistical error $\textup{d}(F^{t}, F^*)$ versus the iteration number $t$, using diagonal thresholding initialization. Panel (b): same plots using perturbation initialization. For both panels (a) and (b), simulations are performed using $d = 5000$, $r = 1$, $k=5$, $\gamma = 4$ and $n = 4000$. Panel (c): plot of estimation error $\textup{d}(\widehat{F}, F^*)$ versus $\frac{k}{n}$, for different values of $(k,n)$ using diagonal thresholding initialization. Each point represents the average over $20$ random instances. The simulation is performed using $d = 5000$, $r = 1$ and $\gamma = 4$.
• Figure 3: Simulations for planted densest subgraph. Panel (a): plots of optimization error $\textup{d} (F^{t}, F^{T} )$ and statistical error $\textup{d}(F^{t}, F^*)$ versus the iteration number $t$, using SVD-based initialization. The simulation is performed using $d = 8000$, $k =2000$, $p = 0.13$ and $q = 0.05$. Panel (b): plot of the probability of successful exact recovery of $F^*$ versus $p d$, for different values of $(d, p)$ using SVD-based initialization. We declare exact recovery if $\textup{d}(\widehat{F}, F^*) \le 2\times 10^{-3}$, and each point represents frequency of exact recovery over $20$ random instances. The simulation is performed with $q = \frac{p}{4}$ and $k = \frac{d}{2}$.
• Figure 4: Simulation results for one-bit matrix completion. Panel (a); plots of OB optimization error $\textup{d} (F^{t}, F^{T} )$ and statistical error $\textup{d}(F^{t}, F^*)$ versus the iteration number $t$, using random initialization. The simulation is performed using $d = 1000$, $r = 3$ and $p =0.5$. Panel (b) plot of per-entry estimation error $\frac{1}{d^2} \textup{d}(\widehat{F}, F^*)$ versus $\frac{r^3}{d^3}$, for different values of $(d, r)$ using random initialization. Each point represents the average over $20$ random instances. The simulation is performed using $p=0.5$ and $\sigma = \frac{0.5r}{d}$.
• Figure 5: Matrix decomposition: plots of optimization error $\textup{d} (F^{t}, F^{T} )$ and statistical error $\textup{d}(F^{t}, F^*)$ versus the iteration number $t$, using (a) SVD-based initialization and (b) random initialization. The simulation is performed using $d = 600$, $r = 5$, $k =100$ and $\sigma = 0.1 \cdot \frac{r}{d}$. Panel (c): plots of the probability of successful exact recovery of $F^*$ versus $\frac{k}{d}$, for different values of $(d, k)$ using SVD-based initialization. We declare exact recovery if $\textup{d}(\widehat{F}, F^*) \le 2\times 10^{-3}$, and each point represents frequency of exact recovery over $20$ random instances. The simulation is performed using $r=6$ and $\sigma=0$.

Theorems & Definitions (30)

• Definition 1: Local descent condition
• Definition 2: Local Lipschitz
• Definition 3: Local smoothness
• Theorem 1
• Theorem 2
• Remark 1
• Corollary 1
• Definition 4: Restricted isometry property
• Corollary 2
• Corollary 3