Improved Analysis of the Accelerated Noisy Power Method with Applications to Decentralized PCA
Pierre Aguié, Mathieu Even, Laurent Massoulié
TL;DR
This work addresses PCA under inexact matrix-vector products and, in particular, decentralized PCA where communication constraints induce noise in matrix-vector products. It develops the Accelerated Noisy Power Method (ANPM) with momentum, proving that acceleration can be preserved under notably milder noise conditions and achieving a worst-case optimal rate of $\tilde{O}(\sqrt{\lambda_k/(\lambda_k - \lambda_{k+1})})$ when optimized with $\beta^* = \lambda_{k+1}^2/4$. The authors extend the analysis to decentralized settings, deriving ADePM which uses gossip-based averaging and momentum to attain accelerated convergence with similar communication costs to non-accelerated methods. Experiments on synthetic data and real decentralized tasks validate the theoretical gains and show that adaptive momentum tuning performs well in practice. Overall, the paper advances the theory and practice of accelerated PCA under realistic noise constraints and decentralized communication budgets, with first-known provably accelerated decentralized PCA results.
Abstract
We analyze the Accelerated Noisy Power Method, an algorithm for Principal Component Analysis in the setting where only inexact matrix-vector products are available, which can arise for instance in decentralized PCA. While previous works have established that acceleration can improve convergence rates compared to the standard Noisy Power Method, these guarantees require overly restrictive upper bounds on the magnitude of the perturbations, limiting their practical applicability. We provide an improved analysis of this algorithm, which preserves the accelerated convergence rate under much milder conditions on the perturbations. We show that our new analysis is worst-case optimal, in the sense that the convergence rate cannot be improved, and that the noise conditions we derive cannot be relaxed without sacrificing convergence guarantees. We demonstrate the practical relevance of our results by deriving an accelerated algorithm for decentralized PCA, which has similar communication costs to non-accelerated methods. To our knowledge, this is the first decentralized algorithm for PCA with provably accelerated convergence.