Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Learning from Biased and Costly Data Sources: Minimax-optimal Data Collection under a Budget

Michael O. Harding, Vikas Singh, Kirthevasan Kandasamy

TL;DR

The paper addresses budgeted, multi-source data collection with heterogeneous source populations. It introduces the effective sample size, grounded in the χ^2-divergence between the target and source mixtures, and shows that maximizing this quantity yields minimax-optimal risk for population mean and group-conditional mean estimation when paired with a post-stratified estimator; it further extends the framework to prediction via importance-weighted ERM. The results establish matching lower and upper bounds up to lower-order terms and demonstrate practical gains through simulations. This provides a principled, cost-aware data collection strategy applicable to medical studies, polling, and other domains where data come from diverse, costly sources.

Abstract

Data collection is a critical component of modern statistical and machine learning pipelines, particularly when data must be gathered from multiple heterogeneous sources to study a target population of interest. In many use cases, such as medical studies or political polling, different sources incur different sampling costs. Observations often have associated group identities (for example, health markers, demographics, or political affiliations) and the relative composition of these groups may differ substantially, both among the source populations and between sources and target population. In this work, we study multi-source data collection under a fixed budget, focusing on the estimation of population means and group-conditional means. We show that naive data collection strategies (e.g. attempting to "match" the target distribution) or relying on standard estimators (e.g. sample mean) can be highly suboptimal. Instead, we develop a sampling plan which maximizes the effective sample size: the total sample size divided by $D_{χ^2}(q\mid\mid\overline{p}) + 1$, where $q$ is the target distribution, $\overline{p}$ is the aggregated source distribution, and $D_{χ^2}$ is the $χ^2$-divergence. We pair this sampling plan with a classical post-stratification estimator and upper bound its risk. We provide matching lower bounds, establishing that our approach achieves the budgeted minimax optimal risk. Our techniques also extend to prediction problems when minimizing the excess risk, providing a principled approach to multi-source learning with costly and heterogeneous data sources.

Learning from Biased and Costly Data Sources: Minimax-optimal Data Collection under a Budget

TL;DR

The paper addresses budgeted, multi-source data collection with heterogeneous source populations. It introduces the effective sample size, grounded in the χ^2-divergence between the target and source mixtures, and shows that maximizing this quantity yields minimax-optimal risk for population mean and group-conditional mean estimation when paired with a post-stratified estimator; it further extends the framework to prediction via importance-weighted ERM. The results establish matching lower and upper bounds up to lower-order terms and demonstrate practical gains through simulations. This provides a principled, cost-aware data collection strategy applicable to medical studies, polling, and other domains where data come from diverse, costly sources.

Abstract

Data collection is a critical component of modern statistical and machine learning pipelines, particularly when data must be gathered from multiple heterogeneous sources to study a target population of interest. In many use cases, such as medical studies or political polling, different sources incur different sampling costs. Observations often have associated group identities (for example, health markers, demographics, or political affiliations) and the relative composition of these groups may differ substantially, both among the source populations and between sources and target population. In this work, we study multi-source data collection under a fixed budget, focusing on the estimation of population means and group-conditional means. We show that naive data collection strategies (e.g. attempting to "match" the target distribution) or relying on standard estimators (e.g. sample mean) can be highly suboptimal. Instead, we develop a sampling plan which maximizes the effective sample size: the total sample size divided by , where is the target distribution, is the aggregated source distribution, and is the -divergence. We pair this sampling plan with a classical post-stratification estimator and upper bound its risk. We provide matching lower bounds, establishing that our approach achieves the budgeted minimax optimal risk. Our techniques also extend to prediction problems when minimizing the excess risk, providing a principled approach to multi-source learning with costly and heterogeneous data sources.
Paper Structure (31 sections, 25 theorems, 101 equations, 3 figures, 1 table)

This paper contains 31 sections, 25 theorems, 101 equations, 3 figures, 1 table.

Table of Contents

  1. Introduction
  2. Multi-source data collection
  3. Model
  4. Environment
  5. Learning with respect to a target group distribution
  6. Multi-source data collection under a budget
  7. Summary of our contributions and techniques
  8. Effective sample size
  9. Lower bound (§ \ref{['sec:me-lbs']})
  10. Upper bound (§\ref{['sec:mean-est']})
  11. Prediction problems (§ \ref{['sec:prediction']})
  12. Empirical evaluation (App. \ref{['sec:experiments']})
  13. Related Work
  14. Sampling techniques
  15. Effective sample size
  16. ...and 16 more sections

Key Result

Theorem 1

The minimax risk eqn:mmriskpm satisfies the following lower bound, where $\bm{n}^\star_T$ is the sampling plan $\bm{n}$ which maximizes $n_{\rm{eff}}(\bm{n}, q)$ subject to the constraint $\bm{c}^\top \bm{n} \leq B$. We have,

Figures (3)

  • Figure 1: Estimated risk based on 100 simulations in each setting. Error regions represent empirical average $\pm$ 2 SE. Row 1: Population mean under $u_K\,$. Row 2: vector of group means. Row 3: Binary classification under $u_K$.
  • Figure 2: Estimated risk based on 100 simulations in each setting. Error regions represent empirical average $\pm$ 2 SE. Row 1: Population mean. Row 2: Binary classification.
  • Figure 3: Estimated risk based on 100 simulations in each setting. Error regions represent empirical average $\pm$ 2 SE. Row 1: Population mean. Row 2: Binary classification.

Theorems & Definitions (44)

  • Example
  • Theorem 1: Informal
  • Theorem 2: Informal
  • Theorem 3: Informal
  • Theorem 4: Informal
  • Theorem 4
  • proof
  • Lemma 1
  • Lemma 2
  • Theorem 5
  • ...and 34 more