Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Statistical-Computational Trade-offs in Tensor PCA and Related Problems via Communication Complexity

Rishabh Dudeja, Daniel Hsu

TL;DR

Computational lower bounds on the run-time of memory bounded algorithms for Tensor PCA are derived using communication complexity and explain why many estimators for these problems use a memory state that is significantly larger than the effective dimensionality of the parameter of interest.

Abstract

Tensor PCA is a stylized statistical inference problem introduced by Montanari and Richard to study the computational difficulty of estimating an unknown parameter from higher-order moment tensors. Unlike its matrix counterpart, Tensor PCA exhibits a statistical-computational gap, i.e., a sample size regime where the problem is information-theoretically solvable but conjectured to be computationally hard. This paper derives computational lower bounds on the run-time of memory bounded algorithms for Tensor PCA using communication complexity. These lower bounds specify a trade-off among the number of passes through the data sample, the sample size, and the memory required by any algorithm that successfully solves Tensor PCA. While the lower bounds do not rule out polynomial-time algorithms, they do imply that many commonly-used algorithms, such as gradient descent and power method, must have a higher iteration count when the sample size is not large enough. Similar lower bounds are obtained for Non-Gaussian Component Analysis, a family of statistical estimation problems in which low-order moment tensors carry no information about the unknown parameter. Finally, stronger lower bounds are obtained for an asymmetric variant of Tensor PCA and related statistical estimation problems. These results explain why many estimators for these problems use a memory state that is significantly larger than the effective dimensionality of the parameter of interest.

Statistical-Computational Trade-offs in Tensor PCA and Related Problems via Communication Complexity

TL;DR

Computational lower bounds on the run-time of memory bounded algorithms for Tensor PCA are derived using communication complexity and explain why many estimators for these problems use a memory state that is significantly larger than the effective dimensionality of the parameter of interest.

Abstract

Tensor PCA is a stylized statistical inference problem introduced by Montanari and Richard to study the computational difficulty of estimating an unknown parameter from higher-order moment tensors. Unlike its matrix counterpart, Tensor PCA exhibits a statistical-computational gap, i.e., a sample size regime where the problem is information-theoretically solvable but conjectured to be computationally hard. This paper derives computational lower bounds on the run-time of memory bounded algorithms for Tensor PCA using communication complexity. These lower bounds specify a trade-off among the number of passes through the data sample, the sample size, and the memory required by any algorithm that successfully solves Tensor PCA. While the lower bounds do not rule out polynomial-time algorithms, they do imply that many commonly-used algorithms, such as gradient descent and power method, must have a higher iteration count when the sample size is not large enough. Similar lower bounds are obtained for Non-Gaussian Component Analysis, a family of statistical estimation problems in which low-order moment tensors carry no information about the unknown parameter. Finally, stronger lower bounds are obtained for an asymmetric variant of Tensor PCA and related statistical estimation problems. These results explain why many estimators for these problems use a memory state that is significantly larger than the effective dimensionality of the parameter of interest.
Paper Structure (130 sections, 59 theorems, 629 equations, 5 figures)

This paper contains 130 sections, 59 theorems, 629 equations, 5 figures.

Table of Contents

  1. Introduction
  2. Our Contributions
  3. Tensor Principal Component Analysis
  4. Non-Gaussian Component Analysis
  5. Asymmetric Tensor PCA
  6. Canonical Correlation Analysis
  7. Proof Techniques
  8. Related Work
  9. Hardness of Symmetric and Asymmetric Tensor PCA
  10. Hardness of Non-Gaussian Component Analysis
  11. Memory and Communication Lower Bounds for Statistical Inference
  12. Statistical Inference and Computational Constraints
  13. Notation
  14. Important sets:
  15. Linear Algebra:
  16. ...and 115 more sections

Key Result

proposition 1

Let Consider a hypothetical setup in which machine $i \in [m]$ is exceptional, and the data $\bm X_{1:m}$ are sampled independently as follows: Let $\overline{\mathbb{P}}_{0}^{(i)}$ and $\overline{\mathbb{E}}_{0}^{(i)}$ denote the probabilities and expectations in this setup: Also let $\overline{\mathbb{E}}_{0}^{(i)}[\cdot |\cdot ]$ denote conditional expectations in this setup. There is a univ

Figures (5)

  • Figure 1: Template for memory bounded estimation algorithms with resource profile $(N,T,s)$
  • Figure 2: Consequences of Result \ref{['informal:tpca']} for linear memory $k$-TPCA algorithms ($k$ even).
  • Figure 3: Consequences of Result \ref{['informal:atpca']} for $k$-ATPCA algorithms with memory size of $s \asymp d^{\frac{k}{2}}$ bits (left) and $s \asymp d^{b}$ bits for $b < k/2$ (right). The striped triangular region represents the run-time vs. sample size trade-offs ruled out by Result \ref{['informal:atpca']}. The green dot at $(\log_{d}(N) = k/2, \log_{d}(T) = 0)$ in Figure \ref{['fig:atpca-overparametrized']} represents the montanari2014statistical estimator.
  • Figure 4: Template for distributed estimation protocols with parameters $(m,n,b)$.
  • Figure 5: Memory bounded implementation of the iterative algorithm with update rule \ref{['eq:common-iteration-template']}.

Theorems & Definitions (146)

  • definition 1: Memory bounded estimation algorithm with resource profile $(N, T, s)$
  • definition 2: Distributed estimation protocol with parameters $(m, n, b)$
  • remark 1: Deterministic vs. Randomized Distributed Estimation Protocols
  • proof
  • proposition 1
  • proof
  • lemma 1: Linearization
  • proof
  • proposition 2: boucheron2013concentrationhan2018geometric
  • proof
  • ...and 136 more