Predicting Fairness of ML Software Configurations

TL;DR

The paper tackles predicting the fairness of ML hyperparameter configurations for a given dataset, addressing whether a regression model can map hyperparameter traces to group fairness metrics such as $AOD$ and $EOD$. It compares four predictors (including $DNN$ and $SVR$) across five training algorithms on four fairness-sensitive datasets, finding that Tree Regressors and XGBoost deliver the strongest predictive accuracy, frequently achieving $R^2$ values near or above 0.95 in stationary settings. It further investigates robustness to temporal distribution shifts, showing that prediction accuracy under shifts depends on the dataset, protected attribute, and learner, with mixed but sometimes strong robustness for sex as the protected attribute. The results propose a practical framework to guide hyperparameter tuning with fairness considerations, potentially reducing the computational cost of exploring biased configurations, while highlighting limitations and directions for future multi-objective and broader-fairness metrics.

Abstract

This paper investigates the relationships between hyperparameters of machine learning and fairness. Data-driven solutions are increasingly used in critical socio-technical applications where ensuring fairness is important. Rather than explicitly encoding decision logic via control and data structures, the ML developers provide input data, perform some pre-processing, choose ML algorithms, and tune hyperparameters (HPs) to infer a program that encodes the decision logic. Prior works report that the selection of HPs can significantly influence fairness. However, tuning HPs to find an ideal trade-off between accuracy, precision, and fairness has remained an expensive and tedious task. Can we predict fairness of HP configuration for a given dataset? Are the predictions robust to distribution shifts? We focus on group fairness notions and investigate the HP space of 5 training algorithms. We first find that tree regressors and XGBoots significantly outperformed deep neural networks and support vector machines in accurately predicting the fairness of HPs. When predicting the fairness of ML hyperparameters under temporal distribution shift, the tree regressors outperforms the other algorithms with reasonable accuracy. However, the precision depends on the ML training algorithm, dataset, and protected attributes. For example, the tree regressor model was robust for training data shift from 2014 to 2018 on logistic regression and discriminant analysis HPs with sex as the protected attribute; but not for race and other training algorithms. Our method provides a sound framework to efficiently perform fine-tuning of ML training algorithms and understand the relationships between HPs and fairness.

Figures (1)

• Figure 1: Computation Complexities of Tree Regressor vs. XGBoost. Each plot shows one ML training algorithm (Decision Tree Classifier in the left, Logistic Regression in the center, and Support Vector Machine in the right).