Empirical Likelihood-Based Fairness Auditing: Distribution-Free Certification and Flagging

TL;DR

The paper tackles fairness auditing for high-stakes predictions by introducing EL-based Fairness Auditing (ELFA), which delivers distribution-free confidence regions and $p$-values for group disparities without relying on parametric distributions. It develops two certification paths, Empirical Likelihood (EL) and Empirical Euclidean Likelihood (EEL), to test whether disparities vanish and to handle nuisance targets via plug-in estimators, with asymptotic results showing $\ell_{EL}(\epsilon_0) \to \chi^2_m$ and $\ell_{EEL}(\epsilon_0) \to \chi^2_m$. For flagging unfair subgroups, four frameworks using equality and inequality constraints are developed, including one- and two-sided tests that exhibit mixture chi-square limits at boundaries; an ELBH procedure is proposed to control false discovery rate (FDR) when flagging across many groups. Simulation studies demonstrate ELFA’s superior coverage accuracy and dramatic reductions in computation time versus bootstrap, while real-data analysis on COMPAS highlights intersectional disparities, such as higher PPV for African-American males under 25 and under-prediction for Caucasian females, corroborating ELFA’s practical value for transparency and accountability in algorithmic decision-making.

Abstract

Machine learning models in high-stakes applications, such as recidivism prediction and automated personnel selection, often exhibit systematic performance disparities across sensitive subpopulations, raising critical concerns regarding algorithmic bias. Fairness auditing addresses these risks through two primary functions: certification, which verifies adherence to fairness constraints; and flagging, which isolates specific demographic groups experiencing disparate treatment. However, existing auditing techniques are frequently limited by restrictive distributional assumptions or prohibitive computational overhead. We propose a novel empirical likelihood-based (EL) framework that constructs robust statistical measures for model performance disparities. Unlike traditional methods, our approach is non-parametric; the proposed disparity statistics follow asymptotically chi-square or mixed chi-square distributions, ensuring valid inference without assuming underlying data distributions. This framework uses a constrained optimization profile that admits stable numerical solutions, facilitating both large-scale certification and efficient subpopulation discovery. Empirically, the EL methods outperform bootstrap-based approaches, yielding coverage rates closer to nominal levels while reducing computational latency by several orders of magnitude. We demonstrate the practical utility of this framework on the COMPAS dataset, where it successfully flags intersectional biases, specifically identifying a significantly higher positive prediction rate for African-American males under 25 and a systemic under-prediction for Caucasian females relative to the population mean.

Figures (9)

• Figure 1: Q-Q plots of $\ell_{EL}({\hbox{\boldmath$\epsilon$}}_0)$ and $\chi^2_m$ under Model \ref{['model1']} and Model \ref{['model2']}, respectively.
• Figure 2: Q-Q plots of $\ell_{EEL}({\hbox{\boldmath$\epsilon$}}_0)$ and $\chi^2_m$ under Model \ref{['model1']} and Model \ref{['model2']}, respectively.
• Figure 3: Power plots under Model \ref{['model3']}: (a) to (d) show results for different sample sizes with $\nu \sim N (0, 1)$. The gray horizontal line in the figure corresponds to $\alpha= 0.05$.
• Figure 4: Power plots under Model \ref{['model3']}: (a) to (d) show results for different sample sizes with $\nu \sim Exp (1) - 1$. The gray horizontal line in the figure corresponds to $\alpha= 0.05$.
• Figure 5: FDR control and statistical power under Model \ref{['model3']}: (a) As $\tau$ increases, the test parameter deviates further from the true value. The FDR, representing the proportion of incorrectly flagged groups, remains below $\alpha$ as shown in the figure, while the power approaches 1. (b) Red circles indicate rejections for the first group, and blue triangles indicate rejections for the second group. As $\tau$ increases and the test parameter deviates further from the true group values, the tests rapidly converge to rejection.