Black-Box $k$-to-$1$-PCA Reductions: Theory and Applications
Arun Jambulapati, Syamantak Kumar, Jerry Li, Shourya Pandey, Ankit Pensia, Kevin Tian
TL;DR
This work develops a theory for black-box deflation-based reductions to compute top-$k$ PCA components using a $1$-PCA oracle. It analyzes two approximation notions, energy-PCA (ePCA) and correlation-PCA (cPCA), showing that ePCA reductions are lossless, while cPCA reductions are lossless only in regimes where $ ext{Delta} imes ext{kappa}_k(oldsymbol{M})^2 \, extless\ Gamma^2$, with a tractable degradation for constant $k$ in the general case. The authors provide a detailed composition framework, including gap-free Wedin decompositions and head-guarantee techniques, to bound parameter blow-up when chaining approximate PCAs, and they apply these insights to obtain robust $k$-PCA algorithms under sub-Gaussian and hypercontractive distributions, as well as online heavy-tailed PCA via Oja’s algorithm. The results yield state-of-the-art robustness and sample-efficiency gains, notably reducing dependence on $k$ and enabling practical, black-box PCA pipelines. Overall, the work offers a cohesive blueprint for designing lossless or near-lossless black-box $k$-PCA reductions with broad statistical applications.
Abstract
The $k$-principal component analysis ($k$-PCA) problem is a fundamental algorithmic primitive that is widely-used in data analysis and dimensionality reduction applications. In statistical settings, the goal of $k$-PCA is to identify a top eigenspace of the covariance matrix of a distribution, which we only have black-box access to via samples. Motivated by these settings, we analyze black-box deflation methods as a framework for designing $k$-PCA algorithms, where we model access to the unknown target matrix via a black-box $1$-PCA oracle which returns an approximate top eigenvector, under two popular notions of approximation. Despite being arguably the most natural reduction-based approach to $k$-PCA algorithm design, such black-box methods, which recursively call a $1$-PCA oracle $k$ times, were previously poorly-understood. Our main contribution is significantly sharper bounds on the approximation parameter degradation of deflation methods for $k$-PCA. For a quadratic form notion of approximation we term ePCA (energy PCA), we show deflation methods suffer no parameter loss. For an alternative well-studied approximation notion we term cPCA (correlation PCA), we tightly characterize the parameter regimes where deflation methods are feasible. Moreover, we show that in all feasible regimes, $k$-cPCA deflation algorithms suffer no asymptotic parameter loss for any constant $k$. We apply our framework to obtain state-of-the-art $k$-PCA algorithms robust to dataset contamination, improving prior work in sample complexity by a $\mathsf{poly}(k)$ factor.