Symmetric Matrix Completion with ReLU Sampling

TL;DR

It is proved that when the matrix factor with a small rank satisfies mild assumptions, the nonconvex objective function is geodesically strongly convex on the quotient manifold in a neighborhood of a planted low-rank matrix.

Abstract

We study the problem of symmetric positive semi-definite low-rank matrix completion (MC) with deterministic entry-dependent sampling. In particular, we consider rectified linear unit (ReLU) sampling, where only positive entries are observed, as well as a generalization to threshold-based sampling. We first empirically demonstrate that the landscape of this MC problem is not globally benign: Gradient descent (GD) with random initialization will generally converge to stationary points that are not globally optimal. Nevertheless, we prove that when the matrix factor with a small rank satisfies mild assumptions, the nonconvex objective function is geodesically strongly convex on the quotient manifold in a neighborhood of a planted low-rank matrix. Moreover, we show that our assumptions are satisfied by a matrix factor with i.i.d. Gaussian entries. Finally, we develop a tailor-designed initialization for GD to solve our studied formulation, which empirically always achieves convergence to the global minima. We also conduct extensive experiments and compare MC methods, investigating convergence and completion performance with respect to initialization, noise level, dimension, and rank.

Figures (9)

• Figure 1: Recovery and convergence performance of GD for solving the MC problem with the uniform $(p=0.2)$ and ReLU sampling in the noiseless case. We apply GD with Gaussian random initialization for solving Problem \ref{['eq:MC']} with the uniform and ReLU sampling, respectively. Then, we plot the gradient norm (i.e., $\|\nabla F(\bm U^{(t)}) \|_F$) and completion error (i.e., $\| \bm U^{(t)}\bm U^{(t)^T} - \bm M \|_F/\|\bm M\|_F$) against number of iterations.
• Figure 2: An illustrative figure on the partition of rows of $\bm U^\star \in \mathbb{R}^{n\times 2}$. We rearrange the rows of $\bm U^\star$ and partition them into 4 blocks, each belonging to different orthants.
• Figure 3: An illustrative figure on \ref{['ass:omegai']} when $r = 2$. Orange pixels denote observed entries, while white pixels denote missing entries.
• Figure 4: A figure on the partition of the plane into 12 equal sectors.
• Figure 5: Convergence and recovery performance of GD for MC with ReLU sampling under different initialization schemes.

Theorems & Definitions (20)

• Theorem 1
• Lemma 1
• Theorem 2
• Example 1: Positive-Threshold Mask
• Theorem 3
• Theorem 4
• Proposition 1
• Proposition 2
• proof : Proof of Theorem \ref{['thm:noiseless:1']}
• proof : Proof of Theorem \ref{['thm:noiseless:2']}