Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

FairLogue: A Toolkit for Intersectional Fairness Analysis in Clinical Machine Learning Models

Nick Souligne, Vignesh Subbian

Abstract

Objective: Algorithmic fairness is essential for equitable and trustworthy machine learning in healthcare. Most fairness tools emphasize single-axis demographic comparisons and may miss compounded disparities affecting intersectional populations. This study introduces Fairlogue, a toolkit designed to operationalize intersectional fairness assessment in observational and counterfactual contexts within clinical settings. Methods: Fairlogue is a Python-based toolkit composed of three components: 1) an observational framework extending demographic parity, equalized odds, and equal opportunity difference to intersectional populations; 2) a counterfactual framework evaluating fairness under treatment-based contexts; and 3) a generalized counterfactual framework assessing fairness under interventions on intersectional group membership. The toolkit was evaluated using electronic health record data from the All of Us Controlled Tier V8 dataset in a glaucoma surgery prediction task using logistic regression with race and gender as protected attributes. Results: Observational analysis identified substantial intersectional disparities despite moderate model performance (AUROC = 0.709; accuracy = 0.651). Intersectional evaluation revealed larger fairness gaps than single-axis analyses, including demographic parity differences of 0.20 and equalized odds true positive and false positive rate gaps of 0.33 and 0.15, respectively. Counterfactual analysis using permutation-based null distributions produced unfairness ("u-value") estimates near zero, suggesting observed disparities were consistent with chance after conditioning on covariates. Conclusion: Fairlogue provides a modular toolkit integrating observational and counterfactual methods for quantifying and evaluating intersectional bias in clinical machine learning workflows.

FairLogue: A Toolkit for Intersectional Fairness Analysis in Clinical Machine Learning Models

Abstract

Objective: Algorithmic fairness is essential for equitable and trustworthy machine learning in healthcare. Most fairness tools emphasize single-axis demographic comparisons and may miss compounded disparities affecting intersectional populations. This study introduces Fairlogue, a toolkit designed to operationalize intersectional fairness assessment in observational and counterfactual contexts within clinical settings. Methods: Fairlogue is a Python-based toolkit composed of three components: 1) an observational framework extending demographic parity, equalized odds, and equal opportunity difference to intersectional populations; 2) a counterfactual framework evaluating fairness under treatment-based contexts; and 3) a generalized counterfactual framework assessing fairness under interventions on intersectional group membership. The toolkit was evaluated using electronic health record data from the All of Us Controlled Tier V8 dataset in a glaucoma surgery prediction task using logistic regression with race and gender as protected attributes. Results: Observational analysis identified substantial intersectional disparities despite moderate model performance (AUROC = 0.709; accuracy = 0.651). Intersectional evaluation revealed larger fairness gaps than single-axis analyses, including demographic parity differences of 0.20 and equalized odds true positive and false positive rate gaps of 0.33 and 0.15, respectively. Counterfactual analysis using permutation-based null distributions produced unfairness ("u-value") estimates near zero, suggesting observed disparities were consistent with chance after conditioning on covariates. Conclusion: Fairlogue provides a modular toolkit integrating observational and counterfactual methods for quantifying and evaluating intersectional bias in clinical machine learning workflows.

Paper Structure

This paper contains 11 sections, 2 equations, 3 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Methods
  4. Component 1: Intersectional Fairness Metrics
  5. Component 2: Counterfactual Intersectional Fairness (with Treatment)
  6. Component 3: Generalized Counterfactual Intersectional Fairness
  7. Toolkit Implementation
  8. Results
  9. Discussion
  10. Conclusion
  11. Acknowledgements

Figures (3)

  • Figure 1: Component 1 workflow for observational intersectional fairness assessment. Input data is used to train a predictive model and compute single-axis and intersectional fairness metrics (demographic parity, equalized odds, equal opportunity difference), producing group-level disparities summaries without causal assumptions.
  • Figure 2: Components 2 and 3 workflows for counterfactual intersectional fairness analysis. Observed dataset evaluated using permutation-based null distributions to compute counterfactual error metrics and unfairness (u-values), enabling assessment of whether disparities persist beyond chance.
  • Figure 3: Plots from Component 3 showing the aggregate disparity metrics with the u-values overlaid in red. Threshold for acceptable unfairness set to 0.1.