Distribution-Specific Auditing For Subgroup Fairness
Daniel Hsu, Jizhou Huang, Brendan Juba
TL;DR
The paper investigates distribution-specific auditing for subgroup fairness, revealing a path from auditing to agnostic learning that yields a PTAS for Gaussian data when restricting to homogeneous halfspaces. It also establishes hardness results for general halfspaces via continuous LWE reductions, showing limits of efficiency under cryptographic assumptions. A practical auditing framework leverages an agnostic-learning oracle to identify highly unfair subgroups, with a concrete PTAS available for the Gaussian homogeneous case. The work emphasizes verification of distributional assumptions to support fairness certificates and outlines directions to broaden applicability beyond Gaussian data.
Abstract
We study the problem of auditing classifiers with the notion of statistical subgroup fairness. Kearns et al. (2018) has shown that the problem of auditing combinatorial subgroups fairness is as hard as agnostic learning. Essentially all work on remedying statistical measures of discrimination against subgroups assumes access to an oracle for this problem, despite the fact that no efficient algorithms are known for it. If we assume the data distribution is Gaussian, or even merely log-concave, then a recent line of work has discovered efficient agnostic learning algorithms for halfspaces. Unfortunately, the reduction of Kearns et al. was formulated in terms of weak, "distribution-free" learning, and thus did not establish a connection for families such as log-concave distributions. In this work, we give positive and negative results on auditing for Gaussian distributions: On the positive side, we present an alternative approach to leverage these advances in agnostic learning and thereby obtain the first polynomial-time approximation scheme (PTAS) for auditing nontrivial combinatorial subgroup fairness: we show how to audit statistical notions of fairness over homogeneous halfspace subgroups when the features are Gaussian. On the negative side, we find that under cryptographic assumptions, no polynomial-time algorithm can guarantee any nontrivial auditing, even under Gaussian feature distributions, for general halfspace subgroups.