On the analysis of optimization with fixed-rank matrices: a quotient geometric view

TL;DR

This work reformulates optimization over fixed-rank matrices as a quotient geometry problem, enabling a quotient-aware Riemannian gradient descent with an explicit update that is invariant to matrix factorization choices. By leveraging a Riemannian preconditioned metric, the authors derive a convergent descent framework under the restricted positive definiteness condition, achieving local linear convergence near the true low-rank matrix in both recovery and completion tasks. The approach yields invariance to balancing of factor matrices and outperforms Euclidean gradient methods in experiments on matrix sensing and completion, while offering competitive or superior performance to existing Riemannian and fixed-rank algorithms. Overall, the quotient-geometry perspective provides a principled, scalable pathway for efficiently solving rank-constrained matrix problems with strong theoretical guarantees and practical impact.

Abstract

We study a type of Riemannian gradient descent (RGD) algorithm, designed through Riemannian preconditioning, for optimization on $\mathcal{M}_k^{m\times n}$ -- the set of $m\times n$ real matrices with a fixed rank $k$. Our analysis is based on a quotient geometric view of $\mathcal{M}_k^{m\times n}$: by identifying this set with the quotient manifold of a two-term product space $\mathbb{R}_*^{m\times k}\times \mathbb{R}_*^{n\times k}$ of matrices with full column rank via matrix factorization, we find an explicit form for the update rule of the RGD algorithm, which leads to a novel approach to analysing their convergence behavior in rank-constrained optimization. We then deduce some interesting properties that reflect how RGD distinguishes from other matrix factorization algorithms such as those based on the Euclidean geometry. In particular, we show that the RGD algorithm is not only faster than Euclidean gradient descent but also does not rely on balancing techniques to ensure its efficiency while the latter does. We further show that this RGD algorithm is guaranteed to solve matrix sensing and matrix completion problems with linear convergence rate under the restricted positive definiteness property. Numerical experiments on matrix sensing and completion are provided to demonstrate these properties.

Figures (6)

• Figure 1: Relations between the product space $\mathbb{R}^{m\times k}_*\times\mathbb{R}^{n\times k}_*$ and the quotient space $\mathcal{M}_{k}^{m\times n}$.
• Figure 2: Landscape of the cost function of the simplest matrix factorization, on the 2D plane.
• Figure 3: Compressed sensing results.
• Figure 4: Gradient and stepsize information of Qprecon RGD (RBB).
• Figure 5: Iteration histories of the algorithms with a balanced and an unbalanced initial point. The size of $M^{\star}$ is $100\times200$, with rank $r^{\star}=3$. The sampling rate $p=0.8$. (a)--(b): test RMSEs by time. (c)--(d): paths of the matrix entries $([X_t]_{1,1}, [X_t]_{2,1})$ of the iterates $\{\pi(\bar{x}_{t})\}$.

Theorems & Definitions (48)

• Example 1: Compressed sensing
• Example 2: Matrix completion
• Definition 3: RPD property 10.1093/imanum/drz061
• Definition 4: mishra2012
• Definition 5: absil2014two
• Proposition 6
• Proposition 7
• Remark 8
• Remark 9
• Lemma 10