Absence of spurious solutions far from ground truth: A low-rank analysis with high-order losses

TL;DR

This paper tackles non-convex matrix sensing in the Burer–Monteiro form by analyzing the optimization landscape under restricted isometry properties (RIP). It proves that critical points far from the ground truth are strict saddles with negative curvature scaling with distance to the ground truth, enabling saddle-escape algorithms to reach neighborhoods of the optimum even from poor initializations. To further shape the landscape without heavy over-parameterization, it introduces high-order losses of the form $f_\lambda^l(X) = f(X) + \lambda f^l(X)$ with even $l$, showing that the resulting Hessian has amplified negative directions at distant points and thus accelerates escape from spurious points. Theoretical results are complemented by simulations demonstrating accelerated saddle escape and more favorable curvature with high-order penalties, suggesting that richer objective functions can mimic benefits of lifting while preserving a smaller search space. Overall, the work provides a step toward a general framework for handling non-convex objectives in machine learning by leveraging the interplay between RIP, distance to ground truth, and higher-order penalties.

Abstract

Matrix sensing problems exhibit pervasive non-convexity, plaguing optimization with a proliferation of suboptimal spurious solutions. Avoiding convergence to these critical points poses a major challenge. This work provides new theoretical insights that help demystify the intricacies of the non-convex landscape. In this work, we prove that under certain conditions, critical points sufficiently distant from the ground truth matrix exhibit favorable geometry by being strict saddle points rather than troublesome local minima. Moreover, we introduce the notion of higher-order losses for the matrix sensing problem and show that the incorporation of such losses into the objective function amplifies the negative curvature around those distant critical points. This implies that increasing the complexity of the objective function via high-order losses accelerates the escape from such critical points and acts as a desirable alternative to increasing the complexity of the optimization problem via over-parametrization. By elucidating key characteristics of the non-convex optimization landscape, this work makes progress towards a comprehensive framework for tackling broader machine learning objectives plagued by non-convexity.

Figures (3)

• Figure 1: The evolution of the objective function and the error between the obtained solution $\hat{X}\hat{X}^T$ and the ground truth $M^*$ during the iterations of the perturbed gradient descent method, with a constant step-size. In both cases, high-order loss functions accelerate the convergence.
• Figure 2: The ratio between the largest and smallest eigenvalue of Hessian at the spurious local minimum $\lambda_{\max}/\lambda_{\min}(\nabla^{2} f^{l}(\hat{X}))$ with respect to $\lambda$ under different size $n$.
• Figure 3: The value of the minimum eigenvalue of the Hessian around saddle points: The first row is for randomly generated Gaussian matrix with $m=20, n=20$, and the second row is for problem \ref{['eq:Yalcin-example']} with $n =21, \epsilon = 0.1$. $\lambda = 0$ (left column), $\lambda = 0.5$ (middle column), $\lambda = 5$ (right column), with x-axis and y-axis as two orthogonal directions from the critical point to the ground truth.

Theorems & Definitions (13)

• Definition 1: RIP
• Lemma 1
• Theorem 1
• Theorem 2: zhang2020many
• Theorem 3
• Theorem 4
• Lemma 2
• Lemma 3
• proof : Proof of Theorem \ref{['thm:l2_nospurious']}
• proof : Proof for Theorem \ref{['thm:small_mstar']}