Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Hidden Convexity of Fair PCA and Fast Solver via Eigenvalue Optimization

Junhui Shen, Aaron J. Davis, Ding Lu, Zhaojun Bai

TL;DR

This work addresses the fairness gap in PCA by introducing FPCA, which minimizes the maximum subgroup reconstruction loss to equalize fidelity across two subgroups. It reveals a hidden convexity by reformulating FPCA as an optimization over the joint numerical range and develops an efficient eigenvalue-optimization solver (EigOpt) that computes an orthogonal projection basis $U\in\mathbb{O}^{n\times r}$. Compared with the traditional SDR-based FPCA, the proposed method achieves up to 8× speedups while maintaining comparable fairness and accuracy, with reconstruction performance close to standard PCA. The approach is practical for real-world datasets and can be extended to more subgroups, offering a scalable solution for fair linear dimensionality reduction in diverse applications.

Abstract

Principal Component Analysis (PCA) is a foundational technique in machine learning for dimensionality reduction of high-dimensional datasets. However, PCA could lead to biased outcomes that disadvantage certain subgroups of the underlying datasets. To address the bias issue, a Fair PCA (FPCA) model was introduced by Samadi et al. (2018) for equalizing the reconstruction loss between subgroups. The semidefinite relaxation (SDR) based approach proposed by Samadi et al. (2018) is computationally expensive even for suboptimal solutions. To improve efficiency, several alternative variants of the FPCA model have been developed. These variants often shift the focus away from equalizing the reconstruction loss. In this paper, we identify a hidden convexity in the FPCA model and introduce an algorithm for convex optimization via eigenvalue optimization. Our approach achieves the desired fairness in reconstruction loss without sacrificing performance. As demonstrated in real-world datasets, the proposed FPCA algorithm runs $8\times$ faster than the SDR-based algorithm, and only at most 85% slower than the standard PCA.

Hidden Convexity of Fair PCA and Fast Solver via Eigenvalue Optimization

TL;DR

This work addresses the fairness gap in PCA by introducing FPCA, which minimizes the maximum subgroup reconstruction loss to equalize fidelity across two subgroups. It reveals a hidden convexity by reformulating FPCA as an optimization over the joint numerical range and develops an efficient eigenvalue-optimization solver (EigOpt) that computes an orthogonal projection basis . Compared with the traditional SDR-based FPCA, the proposed method achieves up to 8× speedups while maintaining comparable fairness and accuracy, with reconstruction performance close to standard PCA. The approach is practical for real-world datasets and can be extended to more subgroups, offering a scalable solution for fair linear dimensionality reduction in diverse applications.

Abstract

Principal Component Analysis (PCA) is a foundational technique in machine learning for dimensionality reduction of high-dimensional datasets. However, PCA could lead to biased outcomes that disadvantage certain subgroups of the underlying datasets. To address the bias issue, a Fair PCA (FPCA) model was introduced by Samadi et al. (2018) for equalizing the reconstruction loss between subgroups. The semidefinite relaxation (SDR) based approach proposed by Samadi et al. (2018) is computationally expensive even for suboptimal solutions. To improve efficiency, several alternative variants of the FPCA model have been developed. These variants often shift the focus away from equalizing the reconstruction loss. In this paper, we identify a hidden convexity in the FPCA model and introduce an algorithm for convex optimization via eigenvalue optimization. Our approach achieves the desired fairness in reconstruction loss without sacrificing performance. As demonstrated in real-world datasets, the proposed FPCA algorithm runs faster than the SDR-based algorithm, and only at most 85% slower than the standard PCA.

Paper Structure

This paper contains 30 sections, 4 theorems, 44 equations, 5 figures, 1 table, 2 algorithms.

Table of Contents

  1. Introduction
  2. Fairness in Machine Learning.
  3. PCA and Fair PCA.
  4. Contributions.
  5. Organization.
  6. Notations.
  7. PCA and fairness issue
  8. Principal Component Analysis.
  9. Fairness issue.
  10. Fair PCA and hidden convexity
  11. Fair PCA model
  12. Fair PCA model.
  13. Fair PCA as trace optimization
  14. The SDR-based algorithm
  15. Hidden convexity
  16. ...and 15 more sections

Key Result

Lemma 3.1

Let $D\in\mathbb{R}^{p\times n}$, $U\in \mathbb{O}^{n \times r}$, and ${\rm loss}_{\!_D}(U)$ be as defined in eq:loss. We have where $H_{\!_D}\in \mathbb{R}^{n \times n}$ is a symmetric matrix given by where $\sigma_i(D)$ denotes the $i$-th largest singular value of $D$.

Figures (5)

  • Figure 1: Top panel: the leading principal components of $M$ by the standard PCA, which captures the maximum variance of $A$, at the expense of $B$, leading to an unfair projection. Bottom panel: The FPCA effectively reduces the imbalance in the variances captured.
  • Figure 2: Geometric illustration of the FPCA \ref{['eq:trminmax']}: The yellow region is the joint numerical range $\mathcal{W}_{r}(H_{\!_A},H_{\!_B})$. Each dashed line is a contour of the 'max' function, i.e., solution of $\max\{y_1,y_2\} = c$ for a given constant $c$. The solid blue line is with $y_1 = y_2$. The star marks the optimal solution $y_*$ of \ref{['eq:trminmax']}.
  • Figure 3: An illustration of the eigenvalue function $\phi(t)$
  • Figure 4: Reconstruction error and loss in Example \ref{['eg:PCAvsFPCA_error']}.
  • Figure 5: Error in Loss Ratio $| \frac{{\rm loss}_{\!_A}}{{\rm loss}_{\!_B}}-1|$

Theorems & Definitions (11)

  • Definition 3.1
  • Lemma 3.1
  • proof
  • Theorem 3.1
  • proof
  • Theorem 4.1
  • proof
  • Lemma 4.1
  • proof
  • Example 5.1
  • ...and 1 more