To Pool or Not To Pool: Analyzing the Regularizing Effects of Group-Fair Training on Shared Models

TL;DR

This work derives group-specific bounds on the generalization error of welfare-centric fair machine learning that benefit from the larger sample size of the majority group, by considering group-specific Rademacher averages over a restricted hypothesis class.

Abstract

In fair machine learning, one source of performance disparities between groups is over-fitting to groups with relatively few training samples. We derive group-specific bounds on the generalization error of welfare-centric fair machine learning that benefit from the larger sample size of the majority group. We do this by considering group-specific Rademacher averages over a restricted hypothesis class, which contains the family of models likely to perform well with respect to a fair learning objective (e.g., a power-mean). Our simulations demonstrate these bounds improve over a naive method, as expected by theory, with particularly significant improvement for smaller group sizes.

Figures (4)

• Figure 1: Visualization of unrestricted class $\mathcal{H}$, theoretical restricted class $\mathcal{H}_{i}^{*}$, and samples of empirical restricted class $\hat{\mathcal{H}}_{i}$ (varying $\bm{z}_{i}$). One possible empirical malfare minimizer $\hat{h}$ (contained by $\hat{\mathcal{H}}_{i}$ and $\mathcal{H}_{i}^{*}$ with high probability), as well as the true malfare minimzer $h^{*}$ (which may fall outside of $\mathcal{H}_{i}^{*}$ or $\hat{\mathcal{H}}_{i}$ due to overfitting to groups other than $i$) are also shown.
• Figure 2: Rademacher average samples in the parameter space of $\hat{\mathcal{H}}_i$ for each group $i \in \{1, 2, 3\}$.
• Figure 3: Average test risk of pooled and separately trained models on three groups (see \ref{['tab:logistic-regression-data']}).
• Figure 4: Generalization error bounds derived from original hypothesis class $\mathcal{H}$ and restricted hypothesis classes $\hat{\mathcal{H}}_i$, compared with shared model $\hat{h}$ train-test gap over 7 independent runs, with quartiles and median trend lines.

Theorems & Definitions (14)

• Definition 1: Power-Mean Malfare
• Definition 2: Rademacher Averages
• Theorem 2: Theoretical Group-Regularized Malfare Bounds
• Theorem 2: Empirical Group-Regularized Malfare Bounds
• Corollary 2: Empirical Malfare Generalization Bounds
• Lemma 2: Convex Optimization for Monte-Carlo Rademacher Averages
• Theorem 2: Theoretical Group-Regularized Malfare Bounds
• proof
• Theorem 2: Empirical Group-Regularized Malfare Bounds
• proof