Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

On the Optimality of the Oja's Algorithm for Online PCA

Xin Liang

TL;DR

It is proved that with high probability Oja’s algorithm performs an efficient, gap-free, global convergence rate to approximate an principal component subspace for any sub-Gaussian distribution.

Abstract

In this paper we analyze the behavior of the Oja's algorithm for online/streaming principal component subspace estimation. It is proved that with high probability it performs an efficient, gap-free, global convergence rate to approximate an principal component subspace for any sub-Gaussian distribution. Moreover, it is the first time to show that the convergence rate, namely the upper bound of the approximation, exactly matches the lower bound of an approximation obtained by the offline/classical PCA up to a constant factor.

On the Optimality of the Oja's Algorithm for Online PCA

TL;DR

It is proved that with high probability Oja’s algorithm performs an efficient, gap-free, global convergence rate to approximate an principal component subspace for any sub-Gaussian distribution.

Abstract

In this paper we analyze the behavior of the Oja's algorithm for online/streaming principal component subspace estimation. It is proved that with high probability it performs an efficient, gap-free, global convergence rate to approximate an principal component subspace for any sub-Gaussian distribution. Moreover, it is the first time to show that the convergence rate, namely the upper bound of the approximation, exactly matches the lower bound of an approximation obtained by the offline/classical PCA up to a constant factor.

Paper Structure

This paper contains 17 sections, 13 theorems, 125 equations, 1 table, 1 algorithm.

Table of Contents

  1. Introduction
  2. Notation
  3. Preliminaries
  4. Canonical Angles between Two Subspaces
  5. Orlicz Norms
  6. Detailed Algorithm and Assumptions
  7. Main Results
  8. Increments of One Iteration
  9. Whole Iteration Process with a Good Initial Guess
  10. Estimation with a Random Initial Guess
  11. Lower Bound and Gap-free Feature
  12. Proofs
  13. Proof of Lemma 3.1
  14. Two probability estimations
  15. Proof of Lemma 3.2
  16. ...and 2 more sections

Key Result

Lemma 2.1

We have for $\mathop{\mathrm{ui}}\nolimits=2,{ \mathop{\mathrm{F}}\nolimits}$ In particular, if $p=q$, then the inequality "$\le$" can be replaced by "$=$".

Theorems & Definitions (21)

  • Definition 2.1: bjorkG1973numerical
  • Lemma 2.1
  • proof
  • Definition 2.2
  • Lemma 2.2: vershynin2012introduction
  • Lemma 3.1
  • Lemma 3.2
  • Lemma 3.3
  • Lemma 3.4: huang2021streaming
  • Theorem 3.1
  • ...and 11 more