The Power of Preconditioning in Overparameterized Low-Rank Matrix Sensing

TL;DR

The paper tackles the problem of recovering a PSD low-rank matrix $M_\\star$ from linear measurements when the true rank is unknown and conditioning may be poor. It introduces ScaledGD(\\lambda), a preconditioned gradient descent on a factorized representation with fixed damping $\\lambda$, achieving fast, near-minimax convergence under Gaussian design from a small random initialization. The main contributions include global convergence in the overparameterized setting with iteration complexity $O(\\log \kappa \\cdot \\log(\\kappa n) + \\log(1/\\varepsilon))$ and sample complexity depending on the true rank $r_\\star$ (not the overparameterized rank), a minimax-optimal error in noisy settings, and robustness to approximate low-rankness. The results are complemented by a detailed analysis decomposing iterates into signal, misalignment, and overparameterization components, and by numerical experiments showing substantial speedups over vanilla GD and strong performance in noisy regimes. Overall, the work demonstrates that carefully designed preconditioning can dramatically accelerate convergence and preserve generalization in overparameterized low-rank matrix sensing, with practical implications for large-scale ill-conditioned problems.

Abstract

We propose $\textsf{ScaledGD($λ$)}$, a preconditioned gradient descent method to tackle the low-rank matrix sensing problem when the true rank is unknown, and when the matrix is possibly ill-conditioned. Using overparametrized factor representations, $\textsf{ScaledGD($λ$)}$ starts from a small random initialization, and proceeds by gradient descent with a specific form of damped preconditioning to combat bad curvatures induced by overparameterization and ill-conditioning. At the expense of light computational overhead incurred by preconditioners, $\textsf{ScaledGD($λ$)}$ is remarkably robust to ill-conditioning compared to vanilla gradient descent ($\textsf{GD}$) even with overprameterization. Specifically, we show that, under the Gaussian design, $\textsf{ScaledGD($λ$)}$ converges to the true low-rank matrix at a constant linear rate after a small number of iterations that scales only logarithmically with respect to the condition number and the problem dimension. This significantly improves over the convergence rate of vanilla $\textsf{GD}$ which suffers from a polynomial dependency on the condition number. Our work provides evidence on the power of preconditioning in accelerating the convergence without hurting generalization in overparameterized learning.

Figures (4)

• Figure 1: Comparison between ScaledGD($\lambda$) and GD. The learning rate of GD has been fine-tuned to achieve fastest convergence for each $\kappa$, while that of ScaledGD($\lambda$) is fixed to $0.3$. The initialization scale $\alpha$ in each case has been fine-tuned so that the final accuracy is $10^{-9}$. The details of the experiment are deferred to Section \ref{['sec:numerical']}.
• Figure 2: Relative reconstruction error versus initialization scale $\alpha$. The slope of the dashed line is approximately $1$.
• Figure 3: Relative reconstruction error versus the number of iterates with different overparameterization rank $r$ for ScaledGD($\lambda$) and $\mathsf{PrecGD}$.
• Figure 4: The relative reconstruction error of ScaledGD($\lambda$) versus the number of iterates for ScaledGD($\lambda$) in the noisy setting, where it is observed that the final error of ScaledGD($\lambda$) approaches the minimax error.

Theorems & Definitions (79)

• Theorem 1: Informal
• Remark 1
• Definition 1: Restricted Isometry Property
• Lemma 1
• Theorem 2
• Theorem 3
• Theorem 4
• Theorem 5
• Remark 2
• Lemma 2