Perturbation Analysis of Randomized SVD and its Applications to Statistics

TL;DR

The paper develops a rigorous perturbation theory for randomized SVD (RSVD) in the presence of noise, deriving high-probability bounds for the distance between exact and RSVD-based singular subspaces in both $\ell_2$ and $\ell_{2,\infty}$ senses. It shows phase-transition behavior governed by the singular-value gap and iteration count $g$, balancing approximation error and estimation error when the observed matrix is signal-plus-noise. The authors then apply these results to statistics-inspired problems, including subspace estimation and community detection in random graphs, matrix completion, and PCA with missing data, obtaining nearly-optimal convergence rates and asymptotic normality for RSVD, even with sketching. The work bridges numerical linear algebra and statistical inference, providing guidance on how to choose sketch size $\widetilde{k}$ and power iterations $g$ to achieve sharp, distributional guarantees in large-scale problems. Extensive simulations and real-data analyses corroborate the phase transitions and normal approximations, underscoring RSVD’s practical viability for high-dimensional inference.

Abstract

Randomized singular value decomposition (RSVD) is a class of computationally efficient algorithms for computing the truncated SVD of large data matrices. Given an $m \times n$ matrix $\widehat{\mathbf M}$, the prototypical RSVD algorithm outputs an approximation of the $k$ leading left singular vectors of $\widehat{\mathbf{M}}$ by computing the SVD of $\widehat{\mathbf{M}} (\widehat{\mathbf M}^{\top} \widehat{\mathbf{M}})^{g} \mathbf G$; here $g \geq 1$ is an integer and $\mathbf G \in \mathbb{R}^{n \times \widetilde{k}}$ is a random Gaussian sketching matrix with $\widetilde{k} \geq k$. In this paper we derive upper bounds for the $\ell_2$ and $\ell_{2,\infty}$ distances between the exact left singular vectors $\widehat{\mathbf{U}}$ of $\widehat{\mathbf{M}}$ and its approximation $\widehat{\mathbf{U}}_g$ (obtained via RSVD), as well as entrywise error bounds when $\widehat{\mathbf{M}}$ is projected onto $\widehat{\mathbf{U}}_g \widehat{\mathbf{U}}_g^{\top}$. These bounds depend on the singular values gap and number of power iterations $g$, and smaller gap requires larger values of $g$ to guarantee the convergences of the $\ell_2$ and $\ell_{2,\infty}$ distances. We apply our theoretical results to settings where $\widehat{\mathbf{M}}$ is an additive perturbation of some unobserved signal matrix $\mathbf{M}$. In particular, we obtain the nearly-optimal convergence rate and asymptotic normality for RSVD on three inference problems, namely, subspace estimation and community detection in random graphs, noisy matrix completion, and PCA with missing data.

Figures (9)

• Figure 1: Phase transition diagram of error rates under the random graph setting with $\widetilde{k} \asymp \log n$ and $n \rho_n \asymp n^{\beta}$. Different regions correspond to different convergence rates of $d_2(\widehat{\mathbf{U}}_g, \mathbf{U})$ or $d_{2 \to \infty}(\widehat{\mathbf{U}}_g, \mathbf{U})$; see Eq. \ref{['rate:simple']} for details. The X- and Y-axes represent $g$ and $\beta$, respectively, and dashed and solid lines represent the thresholds $g = \beta^{-1}$ and $g = 1 + \beta^{-1}$.
• Figure 2: Box plots of error rate for $d_{2}(\widehat{{{\mathbf{U}}}}_g, {{\mathbf{U}}})$ (top panel) and $d_{2,\infty}(\widehat{{{\mathbf{U}}}}_g, {{\mathbf{U}}})$ (bottom panel), where $\widetilde{k} = 5 \log n$ and $1 \leq g \leq 5$. The colors denote different sparsity level, with $\rho_n = 1$ being blue and $\rho_n = 3n^{-1/3}$ being yellow.
• Figure 3: Row-wise fluctuations of $\widehat{\mathbf{U}}_g$ and $\widehat{\mathbf{U}}$ for the two-block SBM with $n = 2000$ and either $\rho_n \asymp 1$ (top panels) or $\rho_n \asymp n^{-1/2}$ (bottom panels). From left to right, scatter plots show the row vectors of $\widehat{\mathbf{U}}_g$ for $g = 1, \dots, 5$ and $\widehat{\mathbf{U}}$. Points are colored according to true community memberships. Red dashed curves represent the 95% empirical confidence ellipses; solid black curves represent the 95% theoretical confidence ellipses.
• Figure 4: The $t$-SNE embeddings of the 68k PBMC gene expressions, projected on the top 50 PCs obtained via RSVD-based PCA (Algorithm \ref{['RSRS:pca']}). Top row (left to right): results for $g = 1$ with $\widetilde{k} = 55$, $100$, $300$, $1000$, and the reference embeddings from zheng2017massively. Bottom row (left to right): results for $g = 2$ with the same sequence of $\widetilde{k}$ values.
• Figure S5: Matrix plots for the true $\mathbf{D}$ (Left), partially observed $\mathbf{D}_{0.8}$ (Middle), and partially observed $\mathbf{D}_{0.4}$ (Right).

Theorems & Definitions (26)

• Remark 1
• Theorem 1
• Theorem 2
• Remark 2: Technical ideas behind the $\ell_{2,\infty}$ bound
• Remark 3: Illustrations of the $r_{2,\infty}$ rate
• Remark 4
• Remark 5
• Corollary 1
• Corollary 2
• Remark 6