Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Counterfactual Fairness with Graph Uncertainty

Davi Valério, Chrysoula Zerva, Mariana Pinto, Ricardo Santos, André Carreiro

TL;DR

This work tackles the brittleness of counterfactual fairness (CF) audits that rely on a single fully specified causal graph by introducing CF with Graph Uncertainty (CF-GU). CF-GU bootstraps a causal discovery algorithm under domain knowledge constraints to generate a bag of plausible DAGs, measures edge and graph uncertainty with normalized entropy $H_e$ and $H_{m G}$ (and $H_{m G_A}$ for the protected attribute's descendants), and propagates this uncertainty to CF metrics such as PSR and NSR with confidence intervals. Empirical results on synthetic data, COMPAS, and Adult show that incorporating graph uncertainty yields more reliable fairness assessments, reduces false certainty, and can reveal biases with high confidence even under limited domain knowledge. The approach provides practitioners with reliability indicators for CF audits and a framework to compare different domain-knowledge assumptions, guiding robust decision-making in fairness evaluation.

Abstract

Evaluating machine learning (ML) model bias is key to building trustworthy and robust ML systems. Counterfactual Fairness (CF) audits allow the measurement of bias of ML models with a causal framework, yet their conclusions rely on a single causal graph that is rarely known with certainty in real-world scenarios. We propose CF with Graph Uncertainty (CF-GU), a bias evaluation procedure that incorporates the uncertainty of specifying a causal graph into CF. CF-GU (i) bootstraps a Causal Discovery algorithm under domain knowledge constraints to produce a bag of plausible Directed Acyclic Graphs (DAGs), (ii) quantifies graph uncertainty with the normalized Shannon entropy, and (iii) provides confidence bounds on CF metrics. Experiments on synthetic data show how contrasting domain knowledge assumptions support or refute audits of CF, while experiments on real-world data (COMPAS and Adult datasets) pinpoint well-known biases with high confidence, even when supplied with minimal domain knowledge constraints.

Counterfactual Fairness with Graph Uncertainty

TL;DR

This work tackles the brittleness of counterfactual fairness (CF) audits that rely on a single fully specified causal graph by introducing CF with Graph Uncertainty (CF-GU). CF-GU bootstraps a causal discovery algorithm under domain knowledge constraints to generate a bag of plausible DAGs, measures edge and graph uncertainty with normalized entropy and (and for the protected attribute's descendants), and propagates this uncertainty to CF metrics such as PSR and NSR with confidence intervals. Empirical results on synthetic data, COMPAS, and Adult show that incorporating graph uncertainty yields more reliable fairness assessments, reduces false certainty, and can reveal biases with high confidence even under limited domain knowledge. The approach provides practitioners with reliability indicators for CF audits and a framework to compare different domain-knowledge assumptions, guiding robust decision-making in fairness evaluation.

Abstract

Evaluating machine learning (ML) model bias is key to building trustworthy and robust ML systems. Counterfactual Fairness (CF) audits allow the measurement of bias of ML models with a causal framework, yet their conclusions rely on a single causal graph that is rarely known with certainty in real-world scenarios. We propose CF with Graph Uncertainty (CF-GU), a bias evaluation procedure that incorporates the uncertainty of specifying a causal graph into CF. CF-GU (i) bootstraps a Causal Discovery algorithm under domain knowledge constraints to produce a bag of plausible Directed Acyclic Graphs (DAGs), (ii) quantifies graph uncertainty with the normalized Shannon entropy, and (iii) provides confidence bounds on CF metrics. Experiments on synthetic data show how contrasting domain knowledge assumptions support or refute audits of CF, while experiments on real-world data (COMPAS and Adult datasets) pinpoint well-known biases with high confidence, even when supplied with minimal domain knowledge constraints.
Paper Structure (14 sections, 6 equations, 8 figures, 4 tables)

This paper contains 14 sections, 6 equations, 8 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Background
  3. Structural Causal Models
  4. Counterfactuals
  5. Counterfactual Fairness
  6. Causal Discovery
  7. Counterfactual Fairness with Graph Uncertainty
  8. Experiments
  9. Synthetic data
  10. COMPAS
  11. Adult
  12. Related Work
  13. Conclusion
  14. Acknowledgments.

Figures (8)

  • Figure 1: Counterfactual Fairness with Graph Uncertainty
  • Figure 2: Ground truth DAG for synthetic experiment.
  • Figure 3: Average variance of score $\mathrm{Avg}_i (\text{Var}(R^i_\text{cf}))$ (bars) and 95% CI (whiskers) across domain knowledge scenarios and classifiers, for the synthetic dataset.
  • Figure 4: Average (bars) and 95% CI (whiskers) of the PSR and NSR across causal worlds for each classifier, for the synthetic dataset.
  • Figure 5: Average variance of score $\mathrm{Avg}_i (\text{Var}(R^i_\text{cf}))$ (bars) and 95% CI (whiskers) across domain knowledge scenarios and classifiers, for the COMPAS dataset
  • ...and 3 more figures