FairBalance: How to Achieve Equalized Odds With Data Pre-processing
Zhe Yu, Joymallya Chakraborty, Tim Menzies
TL;DR
The paper tackles equalized odds fairness in machine learning by analyzing how per-group weighted class distributions influence fairness and proposing a model-agnostic pre-processing method, FairBalance, to balance these distributions. It derives a necessary condition (consistent weighted class ratios across demographics) and a sufficient condition (balanced 1:1 weighted class distribution) to achieve smAOD=0, and shows that the proposed weights satisfy both. Empirical results across eight real-world datasets and a deep learning image task demonstrate that FairBalance reliably improves smAOD (and often AOD) with minimal utility loss and low overhead, outperforming several baseline pre-processing methods. The work emphasizes practical applicability for software engineers and highlights avenues for extending fairness guarantees to more complex settings, including biased labels, noisy attributes, and regression problems.
Abstract
This research seeks to benefit the software engineering society by providing a simple yet effective pre-processing approach to achieve equalized odds fairness in machine learning software. Fairness issues have attracted increasing attention since machine learning software is increasingly used for high-stakes and high-risk decisions. Amongst all the existing fairness notions, this work specifically targets "equalized odds" given its advantage in always allowing perfect classifiers. Equalized odds requires that members of every demographic group do not receive disparate mistreatment. Prior works either optimize for an equalized odds related metric during the learning process like a black-box, or manipulate the training data following some intuition. This work studies the root cause of the violation of equalized odds and how to tackle it. We found that equalizing the class distribution in each demographic group with sample weights is a necessary condition for achieving equalized odds without modifying the normal training process. In addition, an important partial condition for equalized odds (zero average odds difference) can be guaranteed when the class distributions are weighted to be not only equal but also balanced (1:1). Based on these analyses, we proposed FairBalance, a pre-processing algorithm which balances the class distribution in each demographic group by assigning calculated weights to the training data. On eight real-world datasets, our empirical results show that, at low computational overhead, the proposed pre-processing algorithm FairBalance can significantly improve equalized odds without much, if any damage to the utility. FairBalance also outperforms existing state-of-the-art approaches in terms of equalized odds. To facilitate reuse, reproduction, and validation, we made our scripts available at https://github.com/hil-se/FairBalance.