Cost Efficient Fairness Audit Under Partial Feedback

TL;DR

The paper studies fairness auditing of classifiers under partial feedback, where true labels are observed only for positively classified cases, and introduces a pragmatic audit-cost model that accounts for feature and labeling costs via $c_{feat}$ and $c_{lab}$. It analyzes two data-distribution settings—a blackbox model and a mixture of exponential families—proposing two auditing algorithms: RS-Audit for the blackbox case and Exp-Audit for the mixture model. Theoretical results establish lower bounds and near-optimal upper bounds on audit cost, with RS-Audit matching a blackbox lower bound up to log factors and Exp-Audit achieving substantially lower cost under mild assumptions by employing learning-from-truncated-samples and MAP oracles for exponential-family mixtures. Empirically, the methods reduce audit costs by around 50% on real datasets (Adult Income and Law School) and demonstrate favorable performance in synthetic mixture experiments, highlighting practical impact for cost-efficient fairness auditing under partial feedback. The work also connects partial feedback fairness auditing to learning from truncated samples, suggesting promising directions for future theory and algorithms in broader data regimes. Δ = \max_{y,a,a'} \left| \mathbb{P}[f=1|Y=y,A=a] - \mathbb{P}[f=1|Y=y,A=a'] \right| and the partial-feedback setting drive the design of the auditing strategies toward efficient data collection and distribution-aware estimation.

Abstract

We study the problem of auditing the fairness of a given classifier under partial feedback, where true labels are available only for positively classified individuals, (e.g., loan repayment outcomes are observed only for approved applicants). We introduce a novel cost model for acquiring additional labeled data, designed to more accurately reflect real-world costs such as credit assessment, loan processing, and potential defaults. Our goal is to find optimal fairness audit algorithms that are more cost-effective than random exploration and natural baselines. In our work, we consider two audit settings: a black-box model with no assumptions on the data distribution, and a mixture model, where features and true labels follow a mixture of exponential family distributions. In the black-box setting, we propose a near-optimal auditing algorithm under mild assumptions and show that a natural baseline can be strictly suboptimal. In the mixture model setting, we design a novel algorithm that achieves significantly lower audit cost than the black-box case. Our approach leverages prior work on learning from truncated samples and maximum-a-posteriori oracles, and extends known results on spherical Gaussian mixtures to handle exponential family mixtures, which may be of independent interest. Moreover, our algorithms apply to popular fairness metrics including demographic parity, equal opportunity, and equalized odds. Empirically, we demonstrate strong performance of our algorithms on real-world fair classification datasets like Adult Income and Law School, consistently outperforming natural baselines by around 50% in terms of audit cost.

Figures (4)

• Figure 1: From left, the first two plots show the comparison of the Baseline (Algorithm 3, Appendix A) and the RS-Audit (Algorithm \ref{['algo:rejection-sampling-based-basic-algo']}) on the Adult Income dataset. The last two plots show the comparison of RS-Audit and Exp-Audit (Algorithm \ref{['algo:mixture-of-exp-family']}).
• Figure 2: Results on the Adult Income dataset with different classifiers
• Figure 3: Results on the Law School dataset with different classifiers
• Figure 4: Results on the Mixture Model for different choices of $c_{feat}$ and $c_{lab}$.

Theorems & Definitions (53)

• Definition 1: $(\varepsilon,\delta)$-Fairness Audit
• Definition 2: Audit Cost
• Theorem 1
• Theorem 2
• Theorem 3: Lower Bound
• Theorem 4
• Remark 1
• Remark 2
• Remark 3
• Remark 4