Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

SureMap: Simultaneous Mean Estimation for Single-Task and Multi-Task Disaggregated Evaluation

Mikhail Khodak, Lester Mackey, Alexandra Chouldechova, Miroslav Dudík

TL;DR

This work develops a disaggregated evaluation method called SureMap that has high estimation accuracy for both multi-task and single-task disaggregated evaluations of blackbox models and combines maximum a posteriori (MAP) estimation using a well-chosen prior together with cross-validation-free tuning via Stein's unbiased risk estimate (SURE).

Abstract

Disaggregated evaluation -- estimation of performance of a machine learning model on different subpopulations -- is a core task when assessing performance and group-fairness of AI systems. A key challenge is that evaluation data is scarce, and subpopulations arising from intersections of attributes (e.g., race, sex, age) are often tiny. Today, it is common for multiple clients to procure the same AI model from a model developer, and the task of disaggregated evaluation is faced by each customer individually. This gives rise to what we call the multi-task disaggregated evaluation problem, wherein multiple clients seek to conduct a disaggregated evaluation of a given model in their own data setting (task). In this work we develop a disaggregated evaluation method called SureMap that has high estimation accuracy for both multi-task and single-task disaggregated evaluations of blackbox models. SureMap's efficiency gains come from (1) transforming the problem into structured simultaneous Gaussian mean estimation and (2) incorporating external data, e.g., from the AI system creator or from their other clients. Our method combines maximum a posteriori (MAP) estimation using a well-chosen prior together with cross-validation-free tuning via Stein's unbiased risk estimate (SURE). We evaluate SureMap on disaggregated evaluation tasks in multiple domains, observing significant accuracy improvements over several strong competitors.

SureMap: Simultaneous Mean Estimation for Single-Task and Multi-Task Disaggregated Evaluation

TL;DR

This work develops a disaggregated evaluation method called SureMap that has high estimation accuracy for both multi-task and single-task disaggregated evaluations of blackbox models and combines maximum a posteriori (MAP) estimation using a well-chosen prior together with cross-validation-free tuning via Stein's unbiased risk estimate (SURE).

Abstract

Disaggregated evaluation -- estimation of performance of a machine learning model on different subpopulations -- is a core task when assessing performance and group-fairness of AI systems. A key challenge is that evaluation data is scarce, and subpopulations arising from intersections of attributes (e.g., race, sex, age) are often tiny. Today, it is common for multiple clients to procure the same AI model from a model developer, and the task of disaggregated evaluation is faced by each customer individually. This gives rise to what we call the multi-task disaggregated evaluation problem, wherein multiple clients seek to conduct a disaggregated evaluation of a given model in their own data setting (task). In this work we develop a disaggregated evaluation method called SureMap that has high estimation accuracy for both multi-task and single-task disaggregated evaluations of blackbox models. SureMap's efficiency gains come from (1) transforming the problem into structured simultaneous Gaussian mean estimation and (2) incorporating external data, e.g., from the AI system creator or from their other clients. Our method combines maximum a posteriori (MAP) estimation using a well-chosen prior together with cross-validation-free tuning via Stein's unbiased risk estimate (SURE). We evaluate SureMap on disaggregated evaluation tasks in multiple domains, observing significant accuracy improvements over several strong competitors.

Paper Structure

This paper contains 46 sections, 7 theorems, 77 equations, 15 figures, 2 algorithms.

Table of Contents

  1. Introduction
  2. Contributions
  3. Related work
  4. Setup
  5. Setting and baselines
  6. A Gaussian model for disaggregated evaluation
  7. The multi-task setting
  8. Methods
  9. Designing a parameterized estimator
  10. An additive intersectional effects prior
  11. Efficiency and expressivity
  12. Tuning by minimizing expected risk
  13. Single-task SureMap
  14. Multi-task SureMap
  15. Limitations
  16. ...and 31 more sections

Key Result

Lemma B.1

Suppose $\mathbf{\bm y}\sim\mathcal{N}({\bm{\mu}},{\bm{\Sigma}})$ for mean ${\bm{\mu}}\in\mathbb R^d$ and diagonal p.s.d. covariance ${\bm{\Sigma}}\in\mathbb R^{d\times d}$, and consider a function ${\bm{\hat{\mu}}}:\mathbb R^d\to\mathbb R^d$ s.t. for every $i\in[d]$ the function $\hat{\mu}_i$ is al

Figures (15)

  • Figure 1: Single-task SureMap. (For multi-task SureMap see §\ref{['app:computation']}.)
  • Figure 2: Single-task evaluations on Diabetes (top, disaggregating by race, sex, and age), Adult (middle, disaggregating by race, sex, and age), and Common Voice (bottom, disaggregating by sex and age). The MAE is averaged across all groups (left), large groups (center), or small groups (right). Large and small groups are defined as above and below median group size.
  • Figure 3: Multi-task evaluations on SLACS (top, disaggregating by race, sex, and age) and CVC (bottom, disaggregating by sex and age). Left: Performance across different subsampling rates. Right: Multiplicative improvement in MAE over naive estimator on individual tasks; subsampling rate=0.1.
  • Figure 4: Left: Comparison of SureMap variants on SLACS. The SureMap ($\ell$) variant sets to zero the entries of ${\bm{\tau}}$ corresponding to interactions of size $>\ell$ (except for the highest-order interactions). Right: Evaluation of different methods as the interpolation coefficient that defines the CVC tasks is varied.
  • Figure 5: Performance as the number of tasks varies, evaluated on SLACS (left) and CVC (right).
  • ...and 10 more figures

Theorems & Definitions (13)

  • Lemma B.1
  • proof
  • Lemma B.2
  • proof
  • Theorem C.1
  • proof
  • Theorem E.1
  • Proposition E.1: Symmetric SMW Formula
  • proof : Proof of Theorem \ref{['clm:st']}
  • Theorem E.2
  • ...and 3 more