Fairness Implications of Encoding Protected Categorical Attributes

TL;DR

The paper investigates how encoding protected categorical attributes affects accuracy and fairness in ML. It compares one-hot and target encoding, formalizes irreducible and reducible biases, and examines regularization techniques (smoothing and Gaussian noise) to mitigate unfairness, including intersectionality effects. Theoretical analysis shows trade-offs between fairness and accuracy, while experiments on COMPAS and FolkTables demonstrate that regularized target encoding can improve fairness with limited accuracy loss, though intersectional feature engineering can amplify discrimination. The work provides practical guidelines for pre-processing in fair ML and emphasizes reproducibility of results and methods.

Abstract

Past research has demonstrated that the explicit use of protected attributes in machine learning can improve both performance and fairness. Many machine learning algorithms, however, cannot directly process categorical attributes, such as country of birth or ethnicity. Because protected attributes frequently are categorical, they must be encoded as features that can be input to a chosen machine learning algorithm, e.g.\ support vector machines, gradient boosting decision trees or linear models. Thereby, encoding methods influence how and what the machine learning algorithm will learn, affecting model performance and fairness. This work compares the accuracy and fairness implications of the two most well-known encoding methods: \emph{one-hot encoding} and \emph{target encoding}. We distinguish between two types of induced bias that may arise from these encoding methods and may lead to unfair models. The first type, \textit{irreducible bias}, is due to direct group category discrimination, and the second type, \textit{reducible bias}, is due to the large variance in statistically underrepresented groups. We investigate the interaction between categorical encodings and target encoding regularization methods that reduce unfairness. Furthermore, we consider the problem of intersectional unfairness that may arise when machine learning best practices improve performance measures by encoding several categorical attributes into a high-cardinality feature.

Figures (7)

• Figure 1: Comparing one-hot encoding and target encoding regularization (Gaussian noise and smoothing) for the Logistic Regression,Neural Network, and Gradient Boosting classifiers over the test set of the COMPAS dataset. Colored dots regard different regularization parameters: the darker the red the higher the regularization. Different colors imply different fairness metrics. Crossed dots regard one-hot encoding and starred dots are the results of models that exclude the use of the protected attribute.
• Figure 2: Impact of the Gaussian noise regularization parameter $\lambda$ on performance and fairness metrics over the test set of the COMPAS dataset using a Logistic Regression with L1 penalty. In the left image the AUC of the all the protected groups over the regularization hyperparameter. On the right, the equal opportunity fairness, demograpic parity and average absolute oods variation throughout the regularization hyperparameter.
• Figure 3: Equal opportunity fairness implications of encoding categorical protected attributes and their regularization effects on the Compas Dataset. Horizontal lines are the base lines where the protected attribute is not included in the training data. Regularized target encoding does not harm fairness metrics but it can improve predictive performance on this dataset.
• Figure 4: Distribution of the protected attribute categories to be encoded and regularized for the COMPAS data compass. Predominant Ethnic categories are African-American and Caucasian
• Figure 5: Distribution of the intersectional protected attribute Ethnic-Marital to be encoded and regularized for the COMPAS data compass. Predominant categories is categories distribution are African-American Single and Caucassian Single