Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Testing for Causal Fairness

Jiarun Fu, LiZhong Ding, Pengqi Li, Qiuning Wei, Yurong Cheng, Xu Chen

TL;DR

The paper tackles the inadequacy of data-based potential outcomes in high-dimensional fairness analysis by introducing a distribution-based POF and reframing fairness as Distributional Closeness Testing. It defines the counterfactual closeness fairness criterion and develops the NAMMD Treatment Effect (N-TE) statistic, underpinning the Counterfactual Fairness-Closeness Testing (CF-CLOT) procedure with bootstrap-based inference and an asymptotic Gaussian guarantee. By tuning the closeness parameter $\epsilon$, CF-CLOT offers strong, neutral, or weak sensitivity to distributional differences between factual and counterfactual outcomes, enabling nuanced fairness assessment. The approach is validated across real-world tabular datasets and high-dimensional tasks (FER and CRA), showing effective identification of unfair sensitive attributes and competitive performance against SCM-based methods. Overall, CF-CLOT provides a theoretically sound, practically flexible framework for trustworthy fairness testing in complex, high-dimensional settings.

Abstract

Causality is widely used in fairness analysis to prevent discrimination on sensitive attributes, such as genders in career recruitment and races in crime prediction. However, the current data-based Potential Outcomes Framework (POF) often leads to untrustworthy fairness analysis results when handling high-dimensional data. To address this, we introduce a distribution-based POF that transform fairness analysis into Distributional Closeness Testing (DCT) by intervening on sensitive attributes. We define counterfactual closeness fairness as the null hypothesis of DCT, where a sensitive attribute is considered fair if its factual and counterfactual potential outcome distributions are sufficiently close. We introduce the Norm-Adaptive Maximum Mean Discrepancy Treatment Effect (N-TE) as a statistic for measuring distributional closeness and apply DCT using the empirical estimator of NTE, referred to Counterfactual Fairness-CLOseness Testing ($\textrm{CF-CLOT}$). To ensure the trustworthiness of testing results, we establish the testing consistency of N-TE through rigorous theoretical analysis. $\textrm{CF-CLOT}$ demonstrates sensitivity in fairness analysis through the flexibility of the closeness parameter $ε$. Unfair sensitive attributes have been successfully tested by $\textrm{CF-CLOT}$ in extensive experiments across various real-world scenarios, which validate the consistency of the testing.

Testing for Causal Fairness

TL;DR

The paper tackles the inadequacy of data-based potential outcomes in high-dimensional fairness analysis by introducing a distribution-based POF and reframing fairness as Distributional Closeness Testing. It defines the counterfactual closeness fairness criterion and develops the NAMMD Treatment Effect (N-TE) statistic, underpinning the Counterfactual Fairness-Closeness Testing (CF-CLOT) procedure with bootstrap-based inference and an asymptotic Gaussian guarantee. By tuning the closeness parameter , CF-CLOT offers strong, neutral, or weak sensitivity to distributional differences between factual and counterfactual outcomes, enabling nuanced fairness assessment. The approach is validated across real-world tabular datasets and high-dimensional tasks (FER and CRA), showing effective identification of unfair sensitive attributes and competitive performance against SCM-based methods. Overall, CF-CLOT provides a theoretically sound, practically flexible framework for trustworthy fairness testing in complex, high-dimensional settings.

Abstract

Causality is widely used in fairness analysis to prevent discrimination on sensitive attributes, such as genders in career recruitment and races in crime prediction. However, the current data-based Potential Outcomes Framework (POF) often leads to untrustworthy fairness analysis results when handling high-dimensional data. To address this, we introduce a distribution-based POF that transform fairness analysis into Distributional Closeness Testing (DCT) by intervening on sensitive attributes. We define counterfactual closeness fairness as the null hypothesis of DCT, where a sensitive attribute is considered fair if its factual and counterfactual potential outcome distributions are sufficiently close. We introduce the Norm-Adaptive Maximum Mean Discrepancy Treatment Effect (N-TE) as a statistic for measuring distributional closeness and apply DCT using the empirical estimator of NTE, referred to Counterfactual Fairness-CLOseness Testing (). To ensure the trustworthiness of testing results, we establish the testing consistency of N-TE through rigorous theoretical analysis. demonstrates sensitivity in fairness analysis through the flexibility of the closeness parameter . Unfair sensitive attributes have been successfully tested by in extensive experiments across various real-world scenarios, which validate the consistency of the testing.

Paper Structure

This paper contains 31 sections, 8 theorems, 37 equations, 6 figures, 1 algorithm.

Table of Contents

  1. Introduction
  2. Preliminaries
  3. Potential outcomes causal framework
  4. Distributional closeness testing
  5. Problem Formulations and Assumptions
  6. Problem formulations
  7. Assumption
  8. The Proposed Framework
  9. Counterfactual fairness-closeness testing
  10. From the flexibility of $\epsilon$ to the sensitivity of $\textrm{CF-CLOT}$
  11. Strong sensitivity (Counterfactual two-sample testing)
  12. Netural sensitivity
  13. Weak sensitivity
  14. Experiments
  15. Real-world dataset experiment across different models
  16. ...and 16 more sections

Key Result

Theorem 1

If ${\mathrm{N}\text{-}\mathrm{TE}}\left(f_Y(\cdot),do(a_t \rightarrow a_t')\right)=\epsilon$ with $\epsilon\in(0,1]$, we have where and $H_{i, j}=\kappa(\hat{y}_i,\hat{y}_j)+\kappa({\hat{y}}_{a_t',i},{\hat{y}}_{a_t',j})-\kappa(\hat{y}_i,\hat{y}_{a_t',j})-\kappa(\hat{y}_{a_t',i},\hat{y}_j)$ as defined earlier.

Figures (6)

  • Figure 1: Results of CNN
  • Figure 2: Results of LSTM
  • Figure 3: Results of Transformer
  • Figure 4: Accuracy and unfairness result for RAF-DB
  • Figure 5: Accuracy and unfairness result for CRA
  • ...and 1 more figures

Theorems & Definitions (18)

  • Definition 1: Distribution-based POF
  • Definition 2: Counterfactual distribution closeness
  • Definition 3: Counterfactual closeness fairness
  • Definition 4: NAMMD treatment effect (N-TE)
  • Definition 5: Counterfactual fairness-closeness hypothesis
  • Theorem 1: $\text{N-TE}$ asymptotic gaussian distribution.
  • Theorem 2: N-TE Type-I error testing power
  • Theorem 3: N-TE testing consistency
  • Lemma 1
  • Lemma 2
  • ...and 8 more