Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Balanced Filtering via Disclosure-Controlled Proxies

Siqi Deng, Emily Diana, Michael Kearns, Aaron Roth

TL;DR

This work study the problem of collecting a cohort or set that is balanced with respect to sensitive groups when group membership is unavailable or prohibited from use at deployment time and requires that the proxy classification does not reveal significantly more information about the sensitive group membership of any individual example compared to population base rates alone.

Abstract

We study the problem of collecting a cohort or set that is balanced with respect to sensitive groups when group membership is unavailable or prohibited from use at deployment time. Specifically, our deployment-time collection mechanism does not reveal significantly more about the group membership of any individual sample than can be ascertained from base rates alone. To do this, we study a learner that can use a small set of labeled data to train a proxy function that can later be used for this filtering or selection task. We then associate the range of the proxy function with sampling probabilities; given a new example, we classify it using our proxy function and then select it with probability corresponding to its proxy classification. Importantly, we require that the proxy classification does not reveal significantly more information about the sensitive group membership of any individual example compared to population base rates alone (i.e., the level of disclosure should be controlled) and show that we can find such a proxy in a sample- and oracle-efficient manner. Finally, we experimentally evaluate our algorithm and analyze its generalization properties.

Balanced Filtering via Disclosure-Controlled Proxies

TL;DR

This work study the problem of collecting a cohort or set that is balanced with respect to sensitive groups when group membership is unavailable or prohibited from use at deployment time and requires that the proxy classification does not reveal significantly more information about the sensitive group membership of any individual example compared to population base rates alone.

Abstract

We study the problem of collecting a cohort or set that is balanced with respect to sensitive groups when group membership is unavailable or prohibited from use at deployment time. Specifically, our deployment-time collection mechanism does not reveal significantly more about the group membership of any individual sample than can be ascertained from base rates alone. To do this, we study a learner that can use a small set of labeled data to train a proxy function that can later be used for this filtering or selection task. We then associate the range of the proxy function with sampling probabilities; given a new example, we classify it using our proxy function and then select it with probability corresponding to its proxy classification. Importantly, we require that the proxy classification does not reveal significantly more information about the sensitive group membership of any individual example compared to population base rates alone (i.e., the level of disclosure should be controlled) and show that we can find such a proxy in a sample- and oracle-efficient manner. Finally, we experimentally evaluate our algorithm and analyze its generalization properties.
Paper Structure (14 sections, 11 theorems, 28 equations, 3 figures, 4 algorithms)

This paper contains 14 sections, 11 theorems, 28 equations, 3 figures, 4 algorithms.

Table of Contents

  1. Introduction
  2. Related Work
  3. Limitations and Discussion
  4. Model and Preliminaries
  5. Computing Sampling Weights from a Proxy (QP Approach)
  6. Learning an $(\alpha,\beta)$ Proxy
  7. Growing the Convex Hull and Learning a Splitting Function
  8. Decision Tree Meta-Algorithm
  9. Experiments
  10. Data, Hyperparameters, and Compute Time
  11. Results
  12. Discussion and Future Work
  13. Omitted Proofs
  14. Additional Experimental Details

Key Result

Lemma 1

Let $A$ be an $l \times K$ matrix and $U$ be a $1 \times K$ vector with $\frac{1}{K}$ in each entry. There exists a stochastic vector $q$ such that $q A = U$ if and only if $U \in C(A)$.

Figures (3)

  • Figure 1: Trade-off of Disclosure and Balance of Proxies on Communities, Adult, and Marketing
  • Figure 2: Trade-off of Disclosure and Balance for Proxy Models on the Communities, Adult, and Marketing data sets with XGBoost Base Model
  • Figure 3: Difference in accuracy between models trained on filtered and unfiltered data on the Communities, Adult, and Marketing data sets with PRC base model.

Theorems & Definitions (37)

  • Definition 1: Balance
  • Definition 2: $\beta$-Approximate Balance
  • Definition 3: Multi-Class Balance
  • Definition 4: $\beta$-Approximate Multi-Class Balance
  • Remark
  • Definition 5: $\alpha$-Disclosive Proxy
  • Definition 6: $\left(\alpha, \beta \right)$ Proxy
  • Definition 7: Convex Hull boyd2004convex
  • Definition 8: Stochastic Vector Bill86
  • Lemma 1: Inclusion in Convex Hull
  • ...and 27 more