PLS-based approach for fair representation learning

TL;DR

This work introduces Fair Partial Least Squares (Fair PLS) to create low-dimensional representations that are predictive yet approximately independent of sensitive attributes. By augmenting the PLS objective with a fairness regularizer, and extending to kernelized variants via HSIC, the method balances utility and demographic parity, with an optional Equality of Odds extension. The authors provide algorithmic details (NIPALS and kernel forms), extensive experiments across six real-world datasets and multiple prediction models, and show that Fair PLS can outperform vanilla fair PCA in both fairness and predictive performance. The approach is applicable to diverse data modalities, including potential use in large language models, and preserves the interpretability advantages of PLS while enabling fair representations and fair predictions in practical deployments.

Abstract

We revisit the problem of fair representation learning by proposing Fair Partial Least Squares (PLS) components. PLS is widely used in statistics to efficiently reduce the dimension of the data by providing representation tailored for the prediction. We propose a novel method to incorporate fairness constraints in the construction of PLS components. This new algorithm provides a feasible way to construct such features both in the linear and the non linear case using kernel embeddings. The efficiency of our method is evaluated on different datasets, and we prove its superiority with respect to standard fair PCA method.

Figures (8)

• Figure 1: ($\mathcal{B}$) Prediction accuracy vs. disparate impact (DI) using various ML models with the new fair representation as input data. Each point represents the average value from a 7-fold cross-validation and the different colors are for the wide range of $\eta$ used to compute the components.
• Figure 2: (Fair Representation). Comparing the objective functions of the Fair PLS formulation for the representation $r_{\eta} (\mathbf{X})$. The blue line shows result ($\mathcal{A} - 1$) while result ($\mathcal{A} - 2$) is represented by the orange one.
• Figure 3: (Fair Representation). The reconstruction error for the new representation $r_{\eta} (\mathbf{X})$ with respect to the original variables $\mathbf{X}$, showing result $(\mathcal{A} - 3$).
• Figure 4: ($\mathcal{B}$ - Fair Predictions) Similar Figure as \ref{['im:fair_predictions_B1']} for Law School and COMPAS datasets. In this case, we measured the fairness of the predictions made with the new representation in terms of the Equality of Opportunity (EOpp), which is represented versus the Accuracy. EOpp is estimated as the ratio $\hat{P}(c(X) = 1 | S = 0, Y = 1) / \hat{P}(c(X) = 1 | S = 1, Y = 1)$
• Figure 5: Comparison of the covariance with respect to the target $Y$ and the sensitive attribute $S$ between the new representation $r(\mathbf{X})$ obtained via the Vanilla Fair PLS and our proposed formulation. The plots display the mean and standard deviation resulting from a 5-fold cross-validation procedure. For $\tau < 0.2$, $r(\mathbf{X}) \in \mathbb{R}^{1000 \times 0}$; for $0.2 \leq \tau < 0.6$, $r(\mathbf{X}) \in \mathbb{R}^{1000 \times 6}$ and for $0.6 \leq \tau < 1.0$, $r(\mathbf{X}) \in \mathbb{R}^{1000 \times 7}$. The data used for this analysis is described in Appendix B. 1. - Synthetic dataset.

Theorems & Definitions (1)

• Definition 1