Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Low-Precision Streaming PCA

Sanjoy Dasgupta, Syamantak Kumar, Shourya Pandey, Purnamrita Sarkar

TL;DR

An information-theoretic lower bound on the quantization resolution required to achieve a target accuracy for the leading eigenvector is established and empirical evaluations on synthetic streams validate the theoretical findings and demonstrate that low-precision methods closely track the performance of standard Oja's algorithm.

Abstract

Low-precision streaming PCA estimates the top principal component in a streaming setting under limited precision. We establish an information-theoretic lower bound on the quantization resolution required to achieve a target accuracy for the leading eigenvector. We study Oja's algorithm for streaming PCA under linear and nonlinear stochastic quantization. The quantized variants use unbiased stochastic quantization of the weight vector and the updates. Under mild moment and spectral-gap assumptions on the data distribution, we show that a batched version achieves the lower bound up to logarithmic factors under both schemes. This leads to a nearly dimension-free quantization error in the nonlinear quantization setting. Empirical evaluations on synthetic streams validate our theoretical findings and demonstrate that our low-precision methods closely track the performance of standard Oja's algorithm.

Low-Precision Streaming PCA

TL;DR

An information-theoretic lower bound on the quantization resolution required to achieve a target accuracy for the leading eigenvector is established and empirical evaluations on synthetic streams validate the theoretical findings and demonstrate that low-precision methods closely track the performance of standard Oja's algorithm.

Abstract

Low-precision streaming PCA estimates the top principal component in a streaming setting under limited precision. We establish an information-theoretic lower bound on the quantization resolution required to achieve a target accuracy for the leading eigenvector. We study Oja's algorithm for streaming PCA under linear and nonlinear stochastic quantization. The quantized variants use unbiased stochastic quantization of the weight vector and the updates. Under mild moment and spectral-gap assumptions on the data distribution, we show that a batched version achieves the lower bound up to logarithmic factors under both schemes. This leads to a nearly dimension-free quantization error in the nonlinear quantization setting. Empirical evaluations on synthetic streams validate our theoretical findings and demonstrate that our low-precision methods closely track the performance of standard Oja's algorithm.
Paper Structure (29 sections, 22 theorems, 130 equations, 4 figures, 1 table, 2 algorithms)

This paper contains 29 sections, 22 theorems, 130 equations, 4 figures, 1 table, 2 algorithms.

Table of Contents

  1. Introduction
  2. Principal Component Analysis.
  3. Our Contributions.
  4. Problem Setup and Preliminaries
  5. Quantization Schemes and Rounding
  6. Main Results
  7. Lower Bounds
  8. Quantized Batched Oja's Algorithm
  9. Guarantees for Low-Precision Oja's Algorithm
  10. Choosing the Optimal Quantization Parameters
  11. Boosting the Probability of Success
  12. Proof Techniques
  13. Experiments
  14. Conclusion
  15. Utlity Results
  16. ...and 14 more sections

Key Result

Lemma 1

[Lower bound for linear quantization] Let $d > 1$ and $\delta > 0$ such that $\delta^2d\leq 0.5$. Let $\mathcal{V}_L$ denote the set of non-zero quantized vectors $\mathbf{w} \in \mathbb{R}^d$ using the linear quantization scheme eq:quantspace such that $\|\mathbf{w}\| \in [1/2,2]$. Then, $\sup_{\ma

Figures (4)

  • Figure 1: We study the effect of different quantization strategies on mean $\sin^2$-error over 10 runs as the number of samples grows on the $x$ axis. Standard uses $b=n$ batches whereas Batched uses $b=10$ batches. Among the quantization algorithms, we see that in $\sin^2$ error, Standard LQ > Batched LQ and Standard NLQ > Batched NLQ.
  • Figure 2: Variation of $\sin^2$-error with (a) sample size, (b) dimension, and (c) quantization bits.
  • Figure A.1: Variation of $\sin^2$-error with: (a) sample size, (b) dimension, and (c) quantization bits.
  • Figure A.2: Variation of $\sin^2$-error with bits for (a) HAR dataset (b) MNIST dataset.

Theorems & Definitions (46)

  • Lemma 1
  • Lemma 2
  • Theorem 1
  • Remark 1: Matching the Upper and Lower Bounds
  • Remark 2
  • Remark 3: Hyperparameters and eigengap
  • Remark 4: Known $n$ in the learning rate
  • Theorem 2
  • Theorem 3
  • Remark 5
  • ...and 36 more