Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Large Scale High-Dimensional Reduced-Rank Linear Discriminant Analysis

Jocelyn T. Chi

TL;DR

This paper addresses the scalability of Reduced-Rank Linear Discriminant Analysis (RRLDA) in high-dimensional and large-scale settings. It reformulates RRLDA as a matrix least-squares problem and applies the randomized Kaczmarz (RK) method to compute the discriminant subspace efficiently without tuning parameters or explicit regularizers. The authors prove convergence properties and reveal implicit regularization toward the least-norm solution, and they demonstrate favorable accuracy-time trade-offs on the Twenty Newsgroups and OASIS MRI datasets. The work offers a practical, scalable alternative for high-dimensional classification with strong theoretical guarantees and broad applicability to large-scale data analyses.

Abstract

Reduced-rank linear discriminant analysis (RRLDA) is a foundational method of dimension reduction for classification that has been useful in a wide range of applications. The goal is to identify an optimal subspace to project the observations onto that simultaneously maximizes between-group variation while minimizing within-group differences. The solution is straight forward when the number of observations is greater than the number of features but computational difficulties arise in both the high-dimensional setting, where there are more features than there are observations, and when the data are very large. Many works have proposed solutions for the high-dimensional setting and frequently involve additional assumptions or tuning parameters. We propose a fast and simple iterative algorithm for both classical and high-dimensional RRLDA on large data that is free from these additional requirements and that comes with guarantees. We also explain how RRLDA-RK provides implicit regularization towards the least norm solution without explicitly incorporating penalties. We demonstrate our algorithm on real data and highlight some results.

Large Scale High-Dimensional Reduced-Rank Linear Discriminant Analysis

TL;DR

This paper addresses the scalability of Reduced-Rank Linear Discriminant Analysis (RRLDA) in high-dimensional and large-scale settings. It reformulates RRLDA as a matrix least-squares problem and applies the randomized Kaczmarz (RK) method to compute the discriminant subspace efficiently without tuning parameters or explicit regularizers. The authors prove convergence properties and reveal implicit regularization toward the least-norm solution, and they demonstrate favorable accuracy-time trade-offs on the Twenty Newsgroups and OASIS MRI datasets. The work offers a practical, scalable alternative for high-dimensional classification with strong theoretical guarantees and broad applicability to large-scale data analyses.

Abstract

Reduced-rank linear discriminant analysis (RRLDA) is a foundational method of dimension reduction for classification that has been useful in a wide range of applications. The goal is to identify an optimal subspace to project the observations onto that simultaneously maximizes between-group variation while minimizing within-group differences. The solution is straight forward when the number of observations is greater than the number of features but computational difficulties arise in both the high-dimensional setting, where there are more features than there are observations, and when the data are very large. Many works have proposed solutions for the high-dimensional setting and frequently involve additional assumptions or tuning parameters. We propose a fast and simple iterative algorithm for both classical and high-dimensional RRLDA on large data that is free from these additional requirements and that comes with guarantees. We also explain how RRLDA-RK provides implicit regularization towards the least norm solution without explicitly incorporating penalties. We demonstrate our algorithm on real data and highlight some results.
Paper Structure (19 sections, 3 theorems, 26 equations, 1 figure, 2 tables, 1 algorithm)

This paper contains 19 sections, 3 theorems, 26 equations, 1 figure, 2 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Overview
  3. Notation and Abbreviations
  4. Reduced-Rank Linear Discriminant Analysis
  5. Related works on classical linear discriminant analysis
  6. High-dimensional scenarios
  7. Other related works on randomized high-dimensional RRLDA
  8. Computational Complexity of RRLDA
  9. Least Squares Formulation of Reduced Rank Linear Discriminant Analysis
  10. Computational Complexity of Least Squares RRLDA
  11. The Randomized Kaczmarz Method
  12. Large Scale High-Dimensional Reduced-Rank Linear Discriminant Analysis
  13. Implicit regularization via the randomized Kaczmarz method
  14. Computational Complexity of RRLDA-RK
  15. Numerical Experiments
  16. ...and 4 more sections

Key Result

Proposition 1

Let $\mathbf{W}^{\star}$ denote the least norm solution of eqn:ls, and suppose that we select the $i^{th}$ observation $\{\mathbf{y}_{i}, \mathbf{X}_{i}\}$ with probability $p_{i} = \frac{\|\mathbf{x}_{i}\|^{2}}{\|\mathbf{X}\|_{\text{F}}^{2}}$. Let $\kappa(\mathbf{X}) = \frac{\|\mathbf{X}\|^{2}_{\te where $\beta = \frac{1}{\{\sigma^{+}_{\min}(\mathbf{X})\}^{2}}$.

Figures (1)

  • Figure 1: Accuracy (left) and timing (right) across kNN settings (knn1, knn5, knn10) for all methods. Bars indicate median accuracy and time (in log seconds); error bars indicate standard deviations across runs.

Theorems & Definitions (5)

  • Proposition 1
  • proof
  • Corollary 1
  • Proposition 2
  • proof