Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

A Geometric Analysis of PCA

Ayoub El Hanchi, Murat Erdogdu, Chris Maddison

TL;DR

This work establishes a geometric-statistical theory for PCA by treating it as an M-estimator on the Grassmannian. It derives a central limit theorem for the principal-subspace error and the asymptotic distribution of the reconstruction excess risk under reconstruction loss, under mild moment and eigengap assumptions. A pivotal contribution is showing the reconstruction risk is generalized self-concordant along relevant geodesics, which enables both precise asymptotic characterizations and non-asymptotic bounds that recover the asymptotics in the large-sample limit. The results highlight that the key determinant of PCA’s excess risk is a second-order covariance term involving projections of X onto top and bottom eigenvectors, and they provide explicit sample-complexity bounds with Gaussian specializations. These insights connect Grassmannian geometry, self-concordant analysis, and statistical risk in PCA, with potential extensions to broader eigenspace estimation problems.

Abstract

What property of the data distribution determines the excess risk of principal component analysis? In this paper, we provide a precise answer to this question. We establish a central limit theorem for the error of the principal subspace estimated by PCA, and derive the asymptotic distribution of its excess risk under the reconstruction loss. We obtain a non-asymptotic upper bound on the excess risk of PCA that recovers, in the large sample limit, our asymptotic characterization. Underlying our contributions is the following result: we prove that the negative block Rayleigh quotient, defined on the Grassmannian, is generalized self-concordant along geodesics emanating from its minimizer of maximum rotation less than $π/4$.

A Geometric Analysis of PCA

TL;DR

This work establishes a geometric-statistical theory for PCA by treating it as an M-estimator on the Grassmannian. It derives a central limit theorem for the principal-subspace error and the asymptotic distribution of the reconstruction excess risk under reconstruction loss, under mild moment and eigengap assumptions. A pivotal contribution is showing the reconstruction risk is generalized self-concordant along relevant geodesics, which enables both precise asymptotic characterizations and non-asymptotic bounds that recover the asymptotics in the large-sample limit. The results highlight that the key determinant of PCA’s excess risk is a second-order covariance term involving projections of X onto top and bottom eigenvectors, and they provide explicit sample-complexity bounds with Gaussian specializations. These insights connect Grassmannian geometry, self-concordant analysis, and statistical risk in PCA, with potential extensions to broader eigenspace estimation problems.

Abstract

What property of the data distribution determines the excess risk of principal component analysis? In this paper, we provide a precise answer to this question. We establish a central limit theorem for the error of the principal subspace estimated by PCA, and derive the asymptotic distribution of its excess risk under the reconstruction loss. We obtain a non-asymptotic upper bound on the excess risk of PCA that recovers, in the large sample limit, our asymptotic characterization. Underlying our contributions is the following result: we prove that the negative block Rayleigh quotient, defined on the Grassmannian, is generalized self-concordant along geodesics emanating from its minimizer of maximum rotation less than .
Paper Structure (30 sections, 19 theorems, 150 equations)

This paper contains 30 sections, 19 theorems, 150 equations.

Table of Contents

  1. Introduction
  2. Problem setup & Background
  3. A redundancy in the parametrization.
  4. Background on the Grassmann manifold
  5. Asymptotic characterization
  6. Self-concordance of the block Rayleigh quotient
  7. Non-asymptotic bound
  8. Discussion
  9. Further background on the Grassmannian
  10. Parallel transport.
  11. Gradients and Hessians.
  12. Higher order derivatives.
  13. Taylor expansions.
  14. Lipschitz continuous derivatives.
  15. Analysis of the block Rayleigh quotient
  16. ...and 15 more sections

Key Result

Theorem 1

Assume that $\lambda_{k} > \lambda_{k+1}$, $\mathop{\mathrm{E}}\nolimits\lbrack\lVert X\rVert_2^2\rbrack$ is finite, and for all $i, s \in [d-k]$ and $j, t \in [k]$, is finite. Define $\delta_{ij} \vcentcolon= \lambda_j - \lambda_{k+i}$. Then as $n \to \infty$, the following holds.

Theorems & Definitions (38)

  • Theorem 1
  • Corollary 1
  • Remark 1: Empirical projectors
  • Example 1: Spiked covariance model
  • Remark 2: Generalized PCA
  • Proposition 1: Generalized self-concordance of the block Rayleigh quotient
  • Corollary 2
  • Theorem 2
  • Remark 3: Variance parameters
  • Example 2: Gaussian model
  • ...and 28 more