Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Rethinking PCA Through Duality

Jan Quan, Johan Suykens, Panagiotis Patrinos

TL;DR

This work recasts principal component analysis (PCA) within the difference-of-convex (DC) duality framework to derive new PCA formulations and kernelizable duals. It establishes three novel DC dual pairs, proves kernelizability and out-of-sample applicability for duals when the primal term is unitarily invariant, and reveals that simultaneous iteration aligns with a DC algorithm for variance maximization, connecting optimization and numerical linear algebra. A kernelizable DC dual for a robust kernel PCA based on $\ell_1$ reconstruction errors is introduced and linked to iteratively reweighted least squares (IRLS). Empirically, simple first-order methods outperform traditional solvers on the right formulations, enabling scalable PCA variants for large datasets and robust settings with outliers, with practical implications for kernel methods and deep learning architectures.

Abstract

Motivated by the recently shown connection between self-attention and (kernel) principal component analysis (PCA), we revisit the fundamentals of PCA. Using the difference-of-convex (DC) framework, we present several novel formulations and provide new theoretical insights. In particular, we show the kernelizability and out-of-sample applicability for a PCA-like family of problems. Moreover, we uncover that simultaneous iteration, which is connected to the classical QR algorithm, is an instance of the difference-of-convex algorithm (DCA), offering an optimization perspective on this longstanding method. Further, we describe new algorithms for PCA and empirically compare them with state-of-the-art methods. Lastly, we introduce a kernelizable dual formulation for a robust variant of PCA that minimizes the $l_1$ deviation of the reconstruction errors.

Rethinking PCA Through Duality

TL;DR

This work recasts principal component analysis (PCA) within the difference-of-convex (DC) duality framework to derive new PCA formulations and kernelizable duals. It establishes three novel DC dual pairs, proves kernelizability and out-of-sample applicability for duals when the primal term is unitarily invariant, and reveals that simultaneous iteration aligns with a DC algorithm for variance maximization, connecting optimization and numerical linear algebra. A kernelizable DC dual for a robust kernel PCA based on reconstruction errors is introduced and linked to iteratively reweighted least squares (IRLS). Empirically, simple first-order methods outperform traditional solvers on the right formulations, enabling scalable PCA variants for large datasets and robust settings with outliers, with practical implications for kernel methods and deep learning architectures.

Abstract

Motivated by the recently shown connection between self-attention and (kernel) principal component analysis (PCA), we revisit the fundamentals of PCA. Using the difference-of-convex (DC) framework, we present several novel formulations and provide new theoretical insights. In particular, we show the kernelizability and out-of-sample applicability for a PCA-like family of problems. Moreover, we uncover that simultaneous iteration, which is connected to the classical QR algorithm, is an instance of the difference-of-convex algorithm (DCA), offering an optimization perspective on this longstanding method. Further, we describe new algorithms for PCA and empirically compare them with state-of-the-art methods. Lastly, we introduce a kernelizable dual formulation for a robust variant of PCA that minimizes the deviation of the reconstruction errors.
Paper Structure (41 sections, 17 theorems, 55 equations, 4 figures, 12 tables, 10 algorithms)

This paper contains 41 sections, 17 theorems, 55 equations, 4 figures, 12 tables, 10 algorithms.

Table of Contents

  1. Introduction
  2. Contributions
  3. Related work
  4. Preliminaries
  5. Notation
  6. Difference-of-convex duality
  7. Difference-of-convex algorithm
  8. Principal Component Analysis
  9. Classical PCA
  10. DC dual formulations
  11. DC algorithms
  12. Robust PCA
  13. Experiments
  14. Limitations
  15. Conclusion and future outlook
  16. ...and 26 more sections

Key Result

Proposition 2.3

Let $G:\mathbb{R}^{d\times s}\to\overline{\mathbb{R}}$, $F:\mathbb{R}^{N\times s}\to\overline{\mathbb{R}}$, be two convex, closed and proper functions, and $X : \mathbb{R}^{d\times s} \to \mathbb{R}^{N\times s}$ a linear mapping. Then, the following pair of primal-dual problems have the same infimum, i.e., strong duality holds. If $W^\star$ is a solution of eq:dc_p, then any $H^\star \in \partial

Figures (4)

  • Figure 1: Classical PCA formulations for a data matrix $X\in\mathbb{R}^{N\times d}$ and $s\leq \mathop{\mathrm{rank}}\nolimits(X)$ principal components. The starred variables denote (not necessarily unique) global minimizers of the associated formulation. A (full) SVD of $X$ is given by $U\mathop{\mathrm{Diag}}\nolimits(\sigma) V^\top$. The arrows $\Leftrightarrow$ denote that the two formulations are 'equivalent', in the sense that one can easily obtain the minimizers of one problem by solving the other.
  • Figure 2: Three fundamental DC dual pairs for PCA.
  • Figure 3: (left) Timings for the problem setting $(2000, 2000, s, 10^{-3})$ as defined in Table \ref{['tab:comp']}, with varying $s$ and one standard deviation error bars. (right) Timings for the problem setting $(2000, d, 30, 10^{-3})$ as defined in Table \ref{['tab:comp']}, with varying $d$ and one standard deviation error bars. In both figures, ZF stands for ZeroFPR and PG stands for proximal gradient.
  • Figure 4: Relative error of the objective value \ref{['n']} or \ref{['o']} for multiple methods with varying tolerance for the stopping error criterion. $f^\star$ denotes the optimal value while ZF is shorthand for ZeroFPR.

Theorems & Definitions (46)

  • Remark 2.1
  • Definition 2.2: (Informal) kernelizability
  • Proposition 2.3: Toland duality toland1979duality
  • Proposition 2.5
  • Theorem 3.1: Fundamental PCA DC pairs
  • Proposition 3.2
  • Proposition 3.3
  • Remark 3.4
  • Proposition 3.5: Kernelizability for unitarily invariant $G$
  • Remark 3.6
  • ...and 36 more