FairLRF: Achieving Fairness through Sparse Low Rank Factorization

TL;DR

FairLRF presents a novel use of sparse low-rank factorization to enhance fairness in deep learning without retraining. By decomposing weight matrices with SVD and applying Hessian-informed, group-specific scoring, it selectively removes bias-inducing components from unitary matrices, achieving improved equalized opportunity and equalized odds across demographic groups. Empirical results on CelebA and Fitzpatrick-17k show FairLRF outperforms conventional LRF methods and fairness-oriented baselines, with robust ablations outlining sensible ranges for sparsity and layer selection. The approach offers a practical, computation-friendly path to fair DL in resource-constrained settings, with publicly available code for replication.

Abstract

As deep learning (DL) techniques become integral to various applications, ensuring model fairness while maintaining high performance has become increasingly critical, particularly in sensitive fields such as medical diagnosis. Although a variety of bias-mitigation methods have been proposed, many rely on computationally expensive debiasing strategies or suffer substantial drops in model accuracy, which limits their practicality in real-world, resource-constrained settings. To address this issue, we propose a fairness-oriented low rank factorization (LRF) framework that leverages singular value decomposition (SVD) to improve DL model fairness. Unlike traditional SVD, which is mainly used for model compression by decomposing and reducing weight matrices, our work shows that SVD can also serve as an effective tool for fairness enhancement. Specifically, we observed that elements in the unitary matrices obtained from SVD contribute unequally to model bias across groups defined by sensitive attributes. Motivated by this observation, we propose a method, named FairLRF, that selectively removes bias-inducing elements from unitary matrices to reduce group disparities, thus enhancing model fairness. Extensive experiments show that our method outperforms conventional LRF methods as well as state-of-the-art fairness-enhancing techniques. Additionally, an ablation study examines how major hyper-parameters may influence the performance of processed models. To the best of our knowledge, this is the first work utilizing SVD not primarily for compression but for fairness enhancement.

Figures (4)

• Figure 1: Performance of truncated SVD with different ranks $k$. The experiment is conducted on a VGG-11 model pre-trained on Fitzpatrick-17k dataset.
• Figure 2: Demonstration of distributions of Hessian values on two groups. Data collected during the FairLRF experiment in Table \ref{['table_fitzpatrick17k_vgg11']} processing the truncated unitary matrix $\hat{U}_{4096 \times 64}$. Green boxes mark an area for comparison.
• Figure 3: Workflow of FairLRF. Truncated SVD demonstrations are originated from swaminathan2020sparse; scoring set image samples come from Fitzpatrick-17k. The importance values during sparse SVD are randomly filled, and numbers like weight matrix sizes are for demonstrative purposes only. The sparse SVD process for the other unitary matrix is omitted for simplicity.
• Figure 4: Average precision and fairness metrics with different settings. Other hyper-parameters keep the same.