Sharp Analysis of Power Iteration for Tensor PCA

TL;DR

We analyze the tensor PCA model in the single-spike regime under random initialization, focusing on the dynamics of power iteration for order $k\ge3$. By coupling the alignment with a one-dimensional polynomial recurrence via a Gaussian-conditioning technique, we establish sharp convergence bounds and reveal that the algorithmic threshold slightly undercuts the conjectured $\Theta(n^{(k-1)/2})$ by polylog factors. The number of iterations to achieve near-perfect alignment scales polylogarithmically with $n$ and $\gamma_n$, and we propose a simple stopping rule that provably yields a signal-aligned iterate. Our results are complemented by extensive simulations, and the Gaussian conditioning framework opens avenues for analyzing similar iterative methods beyond constant-step regimes. Future work includes extending to sub-Gaussian tensors and multi-rank settings, broadening the applicability of these techniques to nonconvex tensor problems.

Abstract

We investigate the power iteration algorithm for the tensor PCA model introduced in Richard and Montanari (2014). Previous work studying the properties of tensor power iteration is either limited to a constant number of iterations, or requires a non-trivial data-independent initialization. In this paper, we move beyond these limitations and analyze the dynamics of randomly initialized tensor power iteration up to polynomially many steps. Our contributions are threefold: First, we establish sharp bounds on the number of iterations required for power method to converge to the planted signal, for a broad range of the signal-to-noise ratios. Second, our analysis reveals that the actual algorithmic threshold for power iteration is smaller than the one conjectured in literature by a polylog(n) factor, where n is the ambient dimension. Finally, we propose a simple and effective stopping criterion for power iteration, which provably outputs a solution that is highly correlated with the true signal. Extensive numerical experiments verify our theoretical results.

Figures (11)

• Figure 1: Comparison of the marginal distributions between $\alpha_t$ and $X_t$, for $t \in \{1, 2, 3, 4\}$. Here, we set $n = 200$, $k = 3$, $\lambda_n = n^{(k - 1) / 2}$, and run tensor power iteration from random initialization on independent datasets for 1000 times. Note that in this figure, the histograms for $\alpha_t$ and $X_t$ overlap a lot with each other (their overlapping regions are indicated by the third color), meaning that the marginal distributions of $\alpha_t$ and $X_t$ are indeed very close.
• Figure 2: Evolution of correlation $|\langle \tilde{\text{\boldmath $v$}}^t, \text{\boldmath $v$} \rangle|$ as a function of the number of iterations $t$. Here, the $x$ axis represents the number of iterations ranging from 0 to 10, and the $y$ axis gives the level of correlation. We repeat the experiment independently for 1000 times for every combination of $(n, k)$, and compute the average correlation.
• Figure 3: Probability of tensor power iteration with random initialization converging to the hidden spike. The $x$ axis stands for $\gamma_n$ and the $y$ axis gives the empirical convergence probability averaged over 1000 independent experiments.
• Figure 4: Illustration of the effectiveness of the stopping rule. The $x$ axis here is the logarithmic of the number of iterations, and the $y$ axis shows the correlation. We independently repeat the experiment 5 times for each setting, and record the correlation along the power iteration trajectory. Here, $T_{\mathsf{stop}}$ is computed using the power iteration iterates and is marked with a circle in the figure.
• Figure 5: Comparison of the marginal distributions between $\alpha_t$ and $X_t$, for $t \in \{1, 2, 3, 4\}$ from left to right. Here, we set $n = 100$, $k = 3$, $\lambda_n = n^{(k - 1) / 2}$, and run tensor power iteration from random initialization on independent datasets for 1000 times.

Theorems & Definitions (17)

• Theorem 2.1
• Remark 2.1
• Theorem 2.2
• Lemma 3.1
• Lemma 3.2
• Proposition 3.1
• Lemma 3.3
• Lemma 3.4
• Lemma 1.1: Tails of the normal distribution
• Lemma 1.2: Bernstein's inequality