Sketching sparse low-rank matrices with near-optimal sample- and time-complexity using message passing

TL;DR

This work develops a sketching framework for recovering an $n_1 \times n_2$ low-rank matrix with $k$-sparse singular vectors from a small, $k$-dependent sketch that is independent of ambient dimensions. It introduces a sparse-parity/DFT sketch and a two-stage, message-passing recovery algorithm that exploits both sparsity and low rank; provides nonasymptotic recovery guarantees for noiseless cases and extends to noisy settings with robust tests and a reduced-sample, higher-cost trade-off. For disjoint supports, the method achieves near-optimal sample complexity $m = \mathcal{O}( r k^2 / \ln k )$ and sub-quadratic runtime, while in the general case, recovery relies on recovering all nonzero entries and then performing eigendecomposition with complexity $\mathcal{O}((rk)^3)$. The approach scales with $k$ rather than $n$, enabling efficient sketching and recovery for very high-dimensional data; simulations corroborate the theory and show favorable comparisons to convex-program based schemes, especially in the sparse regime. The work thus offers a practical, scalable alternative for sparse low-rank matrix sketching with potential impact on sparse PCA, biclustering, and graph-structured data analysis.

Abstract

We consider the problem of recovering an $n_1 \times n_2$ low-rank matrix with $k$-sparse singular vectors from a small number of linear measurements (sketch). We propose a sketching scheme and an algorithm that can recover the singular vectors with high probability, with a sample complexity and running time that both depend only on $k$ and not on the ambient dimensions $n_1$ and $n_2$. Our sketching operator, based on a scheme for compressed sensing by Li et al. and Bakshi et al., uses a combination of a sparse parity check matrix and a partial DFT matrix. Our main contribution is the design and analysis of a two-stage iterative algorithm which recovers the singular vectors by exploiting the simultaneously sparse and low-rank structure of the matrix. We derive a nonasymptotic bound on the probability of exact recovery, which holds for any $n_1\times n_2 $ sparse, low-rank matrix. We also show how the scheme can be adapted to tackle matrices that are approximately sparse and low-rank. The theoretical results are validated by numerical simulations and comparisons with existing schemes that use convex programming for recovery.

Figures (17)

• Figure 1: (a): Pruned bipartite graph for stage A corresponding to $\boldsymbol{H} \in \{0,1\}^{R\times \tilde{n}}$. Here, $\boldsymbol{X} = \lambda\boldsymbol{v}\boldsymbol{v}^T$ with $v_i\neq 0$ for $i\in[4]$ and $v_i=0$ for $5\le i \le n$, and $R=8$. (b)--(c): The graph process that models the recovery of the nonzero entries of $\boldsymbol{X}$. The faded nodes and edges are those that have been peeled off.
• Figure 2: (a): Pruned bipartite graph for stage B with $\boldsymbol{X}=\tilde{\boldsymbol{v}}\tilde{\boldsymbol{v}}^T$, where $\tilde{v}_i\neq 0$ for $i\in [5]$ and $\tilde{v}_i=0$ for $6\le i\le n$. (b)--(d): The graph process that models the recovery of the nonzero entries in $\tilde{\boldsymbol{v}}$ from the nonzero matrix entries recovered in stage A. The faded nodes and edges are those that have been peeled off. At $t=0$, the left node $\tilde{v}_4$ is peeled off based on the recovered nonzero diagonal entry $X_{44}$.
• Figure 3: (a): Pruned bipartite graph for stage B when $\boldsymbol{X}$ is non-symmetric. Here, $\boldsymbol{X}= \sigma \boldsymbol{u} \boldsymbol{v}^T = \tilde{\boldsymbol{u}} \tilde{\boldsymbol{v}}^T$ with $\tilde{u}_j\neq 0$ for $j\in [4]$, $\tilde{u}_j=0$ for $5\le j\le n$, $\tilde{v}_j\neq 0$ for $j\in [3]$ and $\tilde{v}_j=0$ for $4\le j\le n$. (b)--(d): The graph process that models the recovery of $\tilde{\boldsymbol{u}}$ and $\tilde{\boldsymbol{v}}$. At $t=0$, $\tilde{u}_1$ is recovered as 1 and peeled off. Following the recovery of $\tilde{u}_1$, the left nodes $\tilde{v}_1, \tilde{v}_3$ (and $\tilde{u}_4$) can be peeled off too.
• Figure 4: Probability of exact recovery ($y$-axis) versus sketch size $m$ ($x$-axis), for different sparsity levels $k$. The matrices used in all cases have rank $r=3$ and size $n \times n$ with $n=10^4$. The dashed lines indicate the sketch sizes stated in Theorems \ref{['thm:main_result_symm']} and \ref{['thm:nonsym_main_results']}; the dashed lines in (a) and (b) are determined using $\delta =\frac{5}{7}$ and $\delta = \frac{2}{5}$, respectively. Results are averaged over 100 trials.
• Figure 5: Running time (in secs) of the recovery algorithm versus ambient dimension $n$, for different sparsity levels $k$ and sketch sizes $m$. Exact recovery is achieved in every trial. Results are averaged over 50 trials and error bars indicate one standard deviation.

Theorems & Definitions (26)

• Theorem 1: Noiseless symmetric case
• Theorem 2: Noiseless non-symmetric case
• Lemma 3.1
• Lemma 3.2
• Lemma 3.3
• Lemma 3.4
• Lemma 5.1: Tail bound for Binomial
• Definition 5.1: Negative Association (NA)
• Lemma 5.2: Useful properties of NA joagDev1983negativedubhashi1998balls
• Lemma 5.3: Chernoff bound for NA Bernoulli variables dubhashi1998balls