Nonconvex Factorization and Manifold Formulations are Almost Equivalent in Low-rank Matrix Optimization

TL;DR

The paper establishes a rigorous geometric bridge between manifold (embedded) and nonconvex factorization formulations for rank-constrained low-rank matrix optimization by proving a sandwich relation between Riemannian and Euclidean Hessians. This yields an equivalence of FOSPs, SOSPs, and strict saddles across PSD and general settings, and enables transfer of tight geometric properties between formulations. The results extend to both unregularized and regularized factorizations, with spectral bounds that depend on conditioning, and they underpin global optimality and landscape insights in phase retrieval and well-conditioned low-rank problems. The findings explain why factorization and manifold approaches often perform similarly in practice and provide a foundation for further geometric and algorithmic developments in rank-constrained optimization.

Abstract

In this paper, we consider the geometric landscape connection of the widely studied manifold and factorization formulations in low-rank positive semidefinite (PSD) and general matrix optimization. We establish a sandwich relation on the spectrum of Riemannian and Euclidean Hessians at first-order stationary points (FOSPs). As a result of that, we obtain an equivalence on the set of FOSPs, second-order stationary points (SOSPs) and strict saddles between the manifold and the factorization formulations. In addition, we show the sandwich relation can be used to transfer more quantitative geometric properties from one formulation to another. Similarities and differences in the landscape connection under the PSD case and the general case are discussed. To the best of our knowledge, this is the first geometric landscape connection between the manifold and the factorization formulations for handling rank constraints, and it provides a geometric explanation for the similar empirical performance of factorization and manifold approaches in low-rank matrix optimization observed in the literature. In the general low-rank matrix optimization, the landscape connection of two factorization formulations (unregularized and regularized ones) is also provided. By applying these geometric landscape connections, in particular, the sandwich relation, we are able to solve unanswered questions in literature and establish stronger results in the applications on geometric analysis of phase retrieval, well-conditioned low-rank matrix optimization, and the role of regularization in factorization arising from machine learning and signal processing.

Figures (2)

• Figure 1: Relationship of $\mathbb{R}^{p \times r}$, $T_{\mathbf{X}} {\cal M}_{r+}$, ${\mathscr{A}}_{{\rm null}}^{{\mathbf{Y}}}$, ${\mathscr{A}}_{\overline{{\rm null}}}^{{\mathbf{Y}}}$, ${\mathbf{A}}_{\mathbf{Y}}^\xi$, and ${\mathscr{A}}_{\mathbf{Y}}^\xi$.
• Figure 2: Relationship of subspaces involved in two decompositions in Lemma \ref{['lm: decomposition-Rp1p2-general']}. Left hand side: first decomposition in Lemma \ref{['lm: decomposition-Rp1p2-general']} on relationship between $\mathbb{R}^{(p_1+p_2) \times r}$, $T_{\mathbf{X}} {\cal M}_{r}$, ${\mathscr{A}}_{{\rm null}}^{{\mathbf{L}},{\mathbf{R}}}$, ${\mathscr{A}}_{\overline{{\rm null}}}^{{\mathbf{L}},{\mathbf{R}}}$, ${\mathbf{A}}_{{\mathbf{L}},{\mathbf{R}}}^\xi$, and ${\mathscr{A}}_{{\mathbf{L}},{\mathbf{R}}}^\xi$; Right hand side: second decomposition in Lemma \ref{['lm: decomposition-Rp1p2-general']} on relationship between $\mathbb{R}^{(p_1+p_2) \times r}$, $T_{\mathbf{X}} {\cal M}_{r}$, $\widetilde{{\mathscr{A}}}_{{\rm null}}^{\,\,{\mathbf{L}},{\mathbf{R}}}$, $\widetilde{{\mathscr{A}}}_{\overline{{\rm null}}}^{\,\,{\mathbf{L}},{\mathbf{R}}}$, ${\mathbf{A}}_{{\mathbf{L}},{\mathbf{R}}}^{\xi}$, and $\widetilde{{\mathscr{A}}}_{{\mathbf{L}},{\mathbf{R}}}^{\,\,\xi}$.

Theorems & Definitions (35)

• Lemma 1
• Proposition 1: Riemannian and Euclidean Gradients
• Proposition 2: Riemannian and Euclidean Hessians
• Lemma 2
• Lemma 3
• Proposition 3: Decomposition of ${\mathscr{A}}_{\mathbf{Y}}^\xi$ and Bijection Between ${\mathscr{A}}_{\overline{{\rm null}}}^{{\mathbf{Y}}}$ and $T_{\mathbf{X}} {\cal M}_{r+}$
• Theorem 1
• Remark 1
• Remark 2: Connection of ${\mathscr{A}}_{{\rm null}}^{{\mathbf{Y}}}$ and Rotational Invariance of $g({\mathbf{Y}})$
• Corollary 1