Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Lower Bounds for the Convergence of Tensor Power Iteration on Random Overcomplete Models

Yuchen Wu, Kangjie Zhou

TL;DR

This work analyzes tensor power iteration on a random, overcomplete, fourth-order tensor with random initialization. Using a Gaussian conditioning framework inspired by AMP analysis, the authors prove a polynomial-time lower bound: convergence to a true tensor component requires at least polynomially many iterations, refuting claims of logarithmic-time recovery in this regime. They also show that a common CP objective $\mathcal{S}(x)=\|A x\|_4^4$ is strictly increasing along the power iteration path for certain parameter ranges, and provide numerical evidence indicating polynomial-time success in regimes where SOS-based methods are feasible. The results deepen the theoretical understanding of tensor decompositions under overcompleteness and motivate further development of AMP-type tools for non-proportional, long-horizon iterative algorithms.

Abstract

Tensor decomposition serves as a powerful primitive in statistics and machine learning, and has numerous applications in problems such as learning latent variable models or mixture of Gaussians. In this paper, we focus on using power iteration to decompose an overcomplete random tensor. Past work studying the properties of tensor power iteration either requires a non-trivial data-independent initialization, or is restricted to the undercomplete regime. Moreover, several papers implicitly suggest that logarithmically many iterations (in terms of the input dimension) are sufficient for the power method to recover one of the tensor components. Here we present a novel analysis of the dynamics of tensor power iteration from random initialization in the overcomplete regime, where the tensor rank is much greater than its dimension. Surprisingly, we show that polynomially many steps are necessary for convergence of tensor power iteration to any of the true component, which refutes the previous conjecture. On the other hand, our numerical experiments suggest that tensor power iteration successfully recovers tensor components for a broad range of parameters in polynomial time. To further complement our empirical evidence, we prove that a popular objective function for tensor decomposition is strictly increasing along the power iteration path. Our proof is based on the Gaussian conditioning technique, which has been applied to analyze the approximate message passing (AMP) algorithm. The major ingredient of our argument is a conditioning lemma that allows us to generalize AMP-type analysis to non-proportional limit and polynomially many iterations of the power method.

Lower Bounds for the Convergence of Tensor Power Iteration on Random Overcomplete Models

TL;DR

This work analyzes tensor power iteration on a random, overcomplete, fourth-order tensor with random initialization. Using a Gaussian conditioning framework inspired by AMP analysis, the authors prove a polynomial-time lower bound: convergence to a true tensor component requires at least polynomially many iterations, refuting claims of logarithmic-time recovery in this regime. They also show that a common CP objective is strictly increasing along the power iteration path for certain parameter ranges, and provide numerical evidence indicating polynomial-time success in regimes where SOS-based methods are feasible. The results deepen the theoretical understanding of tensor decompositions under overcompleteness and motivate further development of AMP-type tools for non-proportional, long-horizon iterative algorithms.

Abstract

Tensor decomposition serves as a powerful primitive in statistics and machine learning, and has numerous applications in problems such as learning latent variable models or mixture of Gaussians. In this paper, we focus on using power iteration to decompose an overcomplete random tensor. Past work studying the properties of tensor power iteration either requires a non-trivial data-independent initialization, or is restricted to the undercomplete regime. Moreover, several papers implicitly suggest that logarithmically many iterations (in terms of the input dimension) are sufficient for the power method to recover one of the tensor components. Here we present a novel analysis of the dynamics of tensor power iteration from random initialization in the overcomplete regime, where the tensor rank is much greater than its dimension. Surprisingly, we show that polynomially many steps are necessary for convergence of tensor power iteration to any of the true component, which refutes the previous conjecture. On the other hand, our numerical experiments suggest that tensor power iteration successfully recovers tensor components for a broad range of parameters in polynomial time. To further complement our empirical evidence, we prove that a popular objective function for tensor decomposition is strictly increasing along the power iteration path. Our proof is based on the Gaussian conditioning technique, which has been applied to analyze the approximate message passing (AMP) algorithm. The major ingredient of our argument is a conditioning lemma that allows us to generalize AMP-type analysis to non-proportional limit and polynomially many iterations of the power method.
Paper Structure (22 sections, 16 theorems, 180 equations, 3 figures)

This paper contains 22 sections, 16 theorems, 180 equations, 3 figures.

Table of Contents

  1. Introduction
  2. Main results
  3. Preliminaries
  4. Notation
  5. Tensor power iteration
  6. Intuition behind slow convergence
  7. Gaussian conditioning
  8. Proof overview
  9. Proof sketch of Theorem \ref{['thm:negative']}
  10. Numerical experiments
  11. Conclusion
  12. Analysis of tensor power iteration: Proof of Theorem \ref{['thm:negative']}
  13. Proof of Lemma \ref{['lemma:wt']}
  14. Proof of Lemma \ref{['lemma:etat']}
  15. Proof of Lemma \ref{['lemma:norm-of-vt']}
  16. ...and 7 more sections

Key Result

Theorem 1.1

Assume that $k$, $d$ are large enough, and that $k \asymp d^c$ for some $c \in (3/2, 2)$. Then, there exists some $\eta > 0$ that only depends on $c$, such that with high probability the following happens: Tensor power iteration from random initialization fails to identify any true component of $\ma

Figures (3)

  • Figure 1: Success probability of tensor power iteration for varying $k$ and $d$.
  • Figure 2: Success probability of tensor power iteration for varying $d$ and $T$.
  • Figure 3: Theoretical predictions of $\mathcal{S}$ along the power iteration paths versus the corresponding empirical values. Outcomes are averaged over 1000 independent experiments. The error bars reflect the intervals determined by two times the empirical standard deviation.

Theorems & Definitions (20)

  • Theorem 1.1: Slow convergence from random start, informal, see \ref{['thm:negative']}
  • Theorem 1.2: Increasing objective function, informal, see \ref{['thm:increasing']}
  • Proposition 2.1
  • proof
  • Lemma 2.2: Gaussian conditioning
  • Theorem 3.1
  • Remark 3.1
  • Remark 3.2
  • Theorem 3.2
  • Lemma A.1
  • ...and 10 more