From Fragile to Certified: Wasserstein Audits of Group Fairness Under Distribution Shift

TL;DR

This work proposes a Wasserstein distributionally robust framework that certifies worst-case group fairness over a ball of plausible test distributions centered at the empirical law and defines $\varepsilon$-Wasserstein Distributional Fairness ($\varepsilon$-WDF) as the audit target.

Abstract

Group-fairness metrics (e.g., equalized odds) can vary sharply across resamples and are especially brittle under distribution shift, undermining reliable audits. We propose a Wasserstein distributionally robust framework that certifies worst-case group fairness over a ball of plausible test distributions centered at the empirical law. Our formulation unifies common group fairness notions via a generic conditional-probability functional and defines $\varepsilon$-Wasserstein Distributional Fairness ($\varepsilon$-WDF) as the audit target. Leveraging strong duality, we derive tractable reformulations and an efficient estimator (DRUNE) for $\varepsilon$-WDF. We prove feasibility and consistency and establish finite-sample certification guarantees for auditing fairness, along with quantitative bounds under smoothness and margin conditions. Across standard benchmarks and classifiers, $\varepsilon$-WDF delivers stable fairness assessments under distribution shift, providing a principled basis for auditing and certifying group fairness beyond observational data.

Figures (5)

• Figure 1: Sensitivity of group fairness.Red (Sample–Train–Measure): repeatedly subsample 1,000 points (10,000 reps), retrain, recompute fairness. Blue (Fixed-Model–Sample–Measure): train once per dataset, then repeatedly resample 1,000 points to recompute fairness. Large variability across datasets reveals fragility to sampling and measurement instability.
• Figure 2: (a) Density plot comparing empirical and worst-case fairness estimates ($\widehat{f}_{\delta}$) against true fairness values across 10,000 SVM models ($\delta=0.01$, $q=2$). (b) Fairness regularizer $\mathcal{S}_{\delta,q}$ approaching zero as uncertainty parameter $\delta$ decreases. (c) Direct visualization of the gap between worst-case fairness and true fairness values.
• Figure 3: Illustration of the boundary and region sets $\mathcal{R}^{\raisebox{.0\height}{+}}_i$ and $\mathcal{R}^{\raisebox{.0\height}{-}}_i$ defined in Eq. \ref{['eq:generic']}, corresponding to the worst-case distribution described in Proposition \ref{['prp:worst']}. The shaded regions indicate the sets of points within the distance threshold, while the boundaries $\partial^{\raisebox{.0\height}{+}}_i$ and $\partial^{\raisebox{.0\height}{-}}_i$ (for $q \in [1,\infty]$) are shown as level sets of the distance functions.
• Figure 4: Variability of fairness metrics under Scenario 1. The green shaded bands depict the range of Demographic Parity, Equal Opportunity, and Equalized Odds across 10,000 trials, each of which trains a fresh classifier on a new random subsample of 1000 points. The substantial width of these bands illustrates the pronounced fragility of group‐fairness measures to sampling variation.
• Figure 5: Variability of fairness metrics when recomputing on repeated subsamples. A single classifier is trained once on 1000 randomly drawn data points, and then Demographic Parity, Equal Opportunity, and Equalized Odds are recalculated over 10,000 different subsamples of size 1000. The shaded green bands reveal the extent to which fairness assessments fluctuate purely due to sampling variation.

Theorems & Definitions (41)

• Example 1: Equalized Odds
• Remark 1: Robust Optimization
• Definition 1: $\varepsilon$-Wasserstein Distributional Fairness
• Remark 2
• Proposition 1: Feasibility
• Proposition 2: Consistency
• Proposition 3: Shape of Ambiguity Set
• Proposition 4: $\varepsilon\text{-WDF}$ Condition
• Theorem 1: $\varepsilon\text{-WDF}$ Regularizer: $q = \infty$
• Corollary 1: Simplified $\varepsilon\text{-WDF}$ Condition