Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Eigenvalue and Generalized Eigenvalue Problems: Tutorial

Benyamin Ghojogh, Fakhri Karray, Mark Crowley

TL;DR

This tutorial surveys eigenvalue and generalized eigenvalue problems, presenting their formal definitions, optimization-driven derivations, and concrete ML applications such as PCA, kernel SPCA, and Fisher discriminant analysis. It details both quick and rigorously grounded solutions to generalized eigenproblems, and clarifies how standard eigenvalue problems arise as special cases. The discussion highlights the connections to spectral decompositions and the Rayleigh-Ritz framework, and it illustrates practical procedure steps for obtaining eigenpairs in common ML settings. Overall, the work provides a cohesive, technique-focused roadmap for leveraging eigen- and generalized eigenvalue problems in data analysis and dimensionality reduction.

Abstract

This paper is a tutorial for eigenvalue and generalized eigenvalue problems. We first introduce eigenvalue problem, eigen-decomposition (spectral decomposition), and generalized eigenvalue problem. Then, we mention the optimization problems which yield to the eigenvalue and generalized eigenvalue problems. We also provide examples from machine learning, including principal component analysis, kernel supervised principal component analysis, and Fisher discriminant analysis, which result in eigenvalue and generalized eigenvalue problems. Finally, we introduce the solutions to both eigenvalue and generalized eigenvalue problems.

Eigenvalue and Generalized Eigenvalue Problems: Tutorial

TL;DR

This tutorial surveys eigenvalue and generalized eigenvalue problems, presenting their formal definitions, optimization-driven derivations, and concrete ML applications such as PCA, kernel SPCA, and Fisher discriminant analysis. It details both quick and rigorously grounded solutions to generalized eigenproblems, and clarifies how standard eigenvalue problems arise as special cases. The discussion highlights the connections to spectral decompositions and the Rayleigh-Ritz framework, and it illustrates practical procedure steps for obtaining eigenpairs in common ML settings. Overall, the work provides a cohesive, technique-focused roadmap for leveraging eigen- and generalized eigenvalue problems in data analysis and dimensionality reduction.

Abstract

This paper is a tutorial for eigenvalue and generalized eigenvalue problems. We first introduce eigenvalue problem, eigen-decomposition (spectral decomposition), and generalized eigenvalue problem. Then, we mention the optimization problems which yield to the eigenvalue and generalized eigenvalue problems. We also provide examples from machine learning, including principal component analysis, kernel supervised principal component analysis, and Fisher discriminant analysis, which result in eigenvalue and generalized eigenvalue problems. Finally, we introduce the solutions to both eigenvalue and generalized eigenvalue problems.

Paper Structure

This paper contains 35 sections, 1 theorem, 107 equations.

Table of Contents

  1. Introduction
  2. Introducing Eigenvalue and Generalized Eigenvalue Problems
  3. Eigenvalue Problem
  4. Generalized Eigenvalue Problem
  5. Eigenvalue Optimization
  6. Optimization Form 1
  7. Optimization Form 2
  8. Optimization Form 3
  9. Optimization Form 4
  10. Optimization Form 5
  11. Generalized Eigenvalue Optimization
  12. Optimization Form 1
  13. Optimization Form 2
  14. Optimization Form 3
  15. Examples for the Optimization Problems
  16. ...and 20 more sections

Key Result

Proposition 1

In both complete and incomplete SVD of matrix $\boldsymbol{A}$, the left and right singular vectors are the eigenvectors of $\boldsymbol{A}\boldsymbol{A}^\top$ and $\boldsymbol{A}^\top \boldsymbol{A}$, respectively, and the singular values are the square root of eigenvalues of either $\boldsymbol{A}

Theorems & Definitions (2)

  • Proposition 1
  • proof