Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Safe Fairness Guarantees Without Demographics in Classification: Spectral Uncertainty Set Perspective

Ainhize Barrainkua, Santiago Mazuelas, Novi Quadrianto, Jose A. Lozano

TL;DR

SPECTRE is introduced, a minimax-fair method that adjusts the spectrum of a simple Fourier feature mapping and constrains the extent to which the worst-case distribution can deviate from the empirical distribution, and provides a theoretical analysis that derives computable bounds on the worst-case error for both individual groups and the overall population.

Abstract

As automated classification systems become increasingly prevalent, concerns have emerged over their potential to reinforce and amplify existing societal biases. In the light of this issue, many methods have been proposed to enhance the fairness guarantees of classifiers. Most of the existing interventions assume access to group information for all instances, a requirement rarely met in practice. Fairness without access to demographic information has often been approached through robust optimization techniques,which target worst-case outcomes over a set of plausible distributions known as the uncertainty set. However, their effectiveness is strongly influenced by the chosen uncertainty set. In fact, existing approaches often overemphasize outliers or overly pessimistic scenarios, compromising both overall performance and fairness. To overcome these limitations, we introduce SPECTRE, a minimax-fair method that adjusts the spectrum of a simple Fourier feature mapping and constrains the extent to which the worst-case distribution can deviate from the empirical distribution. We perform extensive experiments on the American Community Survey datasets involving 20 states. The safeness of SPECTRE comes as it provides the highest average values on fairness guarantees together with the smallest interquartile range in comparison to state-of-the-art approaches, even compared to those with access to demographic group information. In addition, we provide a theoretical analysis that derives computable bounds on the worst-case error for both individual groups and the overall population, as well as characterizes the worst-case distributions responsible for these extremal performances

Safe Fairness Guarantees Without Demographics in Classification: Spectral Uncertainty Set Perspective

TL;DR

SPECTRE is introduced, a minimax-fair method that adjusts the spectrum of a simple Fourier feature mapping and constrains the extent to which the worst-case distribution can deviate from the empirical distribution, and provides a theoretical analysis that derives computable bounds on the worst-case error for both individual groups and the overall population.

Abstract

As automated classification systems become increasingly prevalent, concerns have emerged over their potential to reinforce and amplify existing societal biases. In the light of this issue, many methods have been proposed to enhance the fairness guarantees of classifiers. Most of the existing interventions assume access to group information for all instances, a requirement rarely met in practice. Fairness without access to demographic information has often been approached through robust optimization techniques,which target worst-case outcomes over a set of plausible distributions known as the uncertainty set. However, their effectiveness is strongly influenced by the chosen uncertainty set. In fact, existing approaches often overemphasize outliers or overly pessimistic scenarios, compromising both overall performance and fairness. To overcome these limitations, we introduce SPECTRE, a minimax-fair method that adjusts the spectrum of a simple Fourier feature mapping and constrains the extent to which the worst-case distribution can deviate from the empirical distribution. We perform extensive experiments on the American Community Survey datasets involving 20 states. The safeness of SPECTRE comes as it provides the highest average values on fairness guarantees together with the smallest interquartile range in comparison to state-of-the-art approaches, even compared to those with access to demographic group information. In addition, we provide a theoretical analysis that derives computable bounds on the worst-case error for both individual groups and the overall population, as well as characterizes the worst-case distributions responsible for these extremal performances
Paper Structure (58 sections, 1 theorem, 22 equations, 24 figures, 16 tables, 1 algorithm)

This paper contains 58 sections, 1 theorem, 22 equations, 24 figures, 16 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Works
  3. Algorithmic Fairness
  4. Measuring Algorithmic Fairness
  5. Fairness-Enhancing Interventions
  6. Algorithmic Fairness in the (Partial) Absence of Sensitive Information
  7. Robust Risk Minimization Based Approaches in Algorithmic Fairness
  8. Spectral Analysis in ML
  9. Theoretical Background
  10. SPECTRE: a SPECTral Uncertainty Set for Robust Estimation
  11. Robust Classifier with Uncertainty Sets Defined by a Feature Mapping
  12. Spectral Uncertainty Set
  13. The Effect of the Scaling Parameter $\sigma$ and the Tunable Confidence $\lambda_0$
  14. SPECTRE
  15. Out-of-Sample Guarantees on Group Error
  16. ...and 43 more sections

Key Result

Theorem 5.1

Let $\Phi$ : $\mathcal{X}$$\times$$\mathcal{Y}$$\rightarrow$$\mathbb{R}^m$ be a feature mapping blind to the sensitive information, $\boldsymbol{\tau} = \mathbb{E}_{p_n}\{ \Phi \}$$\in \mathbb{R}^m$ and $\boldsymbol{\lambda}$$\in \mathbb{R}_{+}^m$. Then $\mathcal{R}^{\uparrow}_{\mathcal{U}}(f^*, p_{ where $c_i = \mathcal{L}(f^*(\boldsymbol{x}^i),y^i) \mathbb{I}\{\boldsymbol{s}^i = s\}$ and $e_i =

Figures (24)

  • Figure 1: The toy dataset. A binary classification problem with 2 non-sensitive features ($X_1$ and $X_2$), a sensitive attribute $S$ and a binary class label $Y$. The toy dataset is comprised of two sensitive groups: a majority ($90\%$) and minority ($10\%$). In both groups the probability of having either value of the class label is 0.5. In the case of the majority group the knowledge of $X_1$ is enough to be able to distinguish the different classes. However, in the case of the minority group the knowledge about both $X_1$ and $X_2$ is required to derive the optimal classification rule. Besides, the standard deviation of minority instances is higher.
  • Figure 2: The effect of the scaling parameter $\sigma$ on (a) the decision boundaries (where training instances are also shown) and (b) overall accuracy and worst-group accuracy for the toy dataset, with different hyperparameter tuning strategies to get the value of $\lambda_0$ (see Section \ref{['sec:experimental_setting']} for descriptions on strategies). In (a), we set $\lambda_0 = 0.1$. Above the graphical representations of the decision boundaries, the value of $\sigma$ under consideration is displayed, along with the corresponding worst-group accuracy and overall accuracy (presented in parentheses, in that order). In (b), the curve represents the average value across 10 runs, and the shaded region denotes the standard deviation.
  • Figure 3: The distribution of the sensitive attribute race for the states of Hawaii and Indiana in the ACSEmployment dataset. The state of Hawaii exhibits greater racial diversity.
  • Figure 4: The distributions of the average overall accuracyand worst group accuracyof SOTA approaches and SPECTRE in (a,c) ACSEmployment and (b,d) ACSIncome datasets, across different collections of 20 randomly selected US states, with (a,b) race as the sensitive attribute and (c,d) considering intersectional groups based on race and gender. SPECTRE consistently outperforms SOTA methods in worst-group accuracy with a minimal overall accuracy reduction with respect to the best performing method (XGB).
  • Figure 5: The predictive performance of MRC on the COMPAS dataset trained and tested individually for the different racial groups for varying value of $\lambda_0$. The worst-group accuracy does not experience significant enhancement with SPECTRE, as the MRC trained individually within the groups struggles to achieve a high level of predictive performance.
  • ...and 19 more figures

Theorems & Definitions (3)

  • Theorem 5.1
  • proof
  • Definition 5.2: Extremal distribution for group $\boldsymbol{s}$