Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Subsampling Suffices for Adaptive Data Analysis

Guy Blanc

TL;DR

This work identifies a remarkably simple set of assumptions under which the queries of broadly applicable class of statistical queries will continue to be representative even when chosen adaptively: the only requirements are that each query takes as input a random subsample and outputs few bits.

Abstract

Ensuring that analyses performed on a dataset are representative of the entire population is one of the central problems in statistics. Most classical techniques assume that the dataset is independent of the analyst's query and break down in the common setting where a dataset is reused for multiple, adaptively chosen, queries. This problem of \emph{adaptive data analysis} was formalized in the seminal works of Dwork et al. (STOC, 2015) and Hardt and Ullman (FOCS, 2014). We identify a remarkably simple set of assumptions under which the queries will continue to be representative even when chosen adaptively: The only requirements are that each query takes as input a random subsample and outputs few bits. This result shows that the noise inherent in subsampling is sufficient to guarantee that query responses generalize. The simplicity of this subsampling-based framework allows it to model a variety of real-world scenarios not covered by prior work. In addition to its simplicity, we demonstrate the utility of this framework by designing mechanisms for two foundational tasks, statistical queries and median finding. In particular, our mechanism for answering the broadly applicable class of statistical queries is both extremely simple and state of the art in many parameter regimes.

Subsampling Suffices for Adaptive Data Analysis

TL;DR

This work identifies a remarkably simple set of assumptions under which the queries of broadly applicable class of statistical queries will continue to be representative even when chosen adaptively: the only requirements are that each query takes as input a random subsample and outputs few bits.

Abstract

Ensuring that analyses performed on a dataset are representative of the entire population is one of the central problems in statistics. Most classical techniques assume that the dataset is independent of the analyst's query and break down in the common setting where a dataset is reused for multiple, adaptively chosen, queries. This problem of \emph{adaptive data analysis} was formalized in the seminal works of Dwork et al. (STOC, 2015) and Hardt and Ullman (FOCS, 2014). We identify a remarkably simple set of assumptions under which the queries will continue to be representative even when chosen adaptively: The only requirements are that each query takes as input a random subsample and outputs few bits. This result shows that the noise inherent in subsampling is sufficient to guarantee that query responses generalize. The simplicity of this subsampling-based framework allows it to model a variety of real-world scenarios not covered by prior work. In addition to its simplicity, we demonstrate the utility of this framework by designing mechanisms for two foundational tasks, statistical queries and median finding. In particular, our mechanism for answering the broadly applicable class of statistical queries is both extremely simple and state of the art in many parameter regimes.
Paper Structure (39 sections, 46 theorems, 164 equations, 5 figures, 1 table)

This paper contains 39 sections, 46 theorems, 164 equations, 5 figures, 1 table.

Table of Contents

  1. Introduction
  2. Main results
  3. Applications
  4. A mechanism for statistical queries
  5. Advantages of our mechanism.
  6. A mechanism for finding approximate medians
  7. Other related work
  8. Technical overview
  9. Sketch of Theorem \ref{['thm:informal']} for the simplest subsampling queries
  10. The remainder of this technical overview
  11. The average leave-many-out KL stability of subsampling queries
  12. Bounding the stability of simple subsampling queries
  13. Handling larger subsampling queries using the concentration of U-statistics
  14. Stability and mutual information
  15. Formal statements of our bias bounds
  16. ...and 24 more sections

Key Result

Theorem 1

Suppose an analyst asks an adaptive sequence of $T$ subsampling queries, each mapping $X^w$ to $Y$, to a sample $\boldsymbol{S} \sim \mathcal{D}^n$. As long as with high probability, all of the queries will have low bias.

Figures (5)

  • Figure 1: A mechanism for answering statistical queries using subsampling.
  • Figure 2: The subsampling mechanism answering approximate median queries.
  • Figure 3: An illustration of why, for $\varphi:X^1 \to Y$, the distribution of $\varphi(S)$ is equivalent to $\varphi(S_{\boldsymbol{i}})$ where $S_1, \ldots, S_k$ are partitions of $S$ and $\boldsymbol{i} \sim \mathrm{Unif}([k])$.
  • Figure 4: An analyst asking an adaptive sequence of subsampling queries.
  • Figure 5: An analyst game.

Theorems & Definitions (99)

  • Definition 1.1: Subsampling query
  • Theorem 1: Subsampling responses have low bias
  • Theorem 2: Subsampling queries reveal little information
  • Theorem 3: Accuracy of our mechanism for answering SQs
  • Definition 1.2: Approximate median
  • Theorem 4: Accuracy of our mechanism for answering approximate median queries
  • Remark 1.3: Design decisions for the mechanism in \ref{['fig:median-mechanism']}
  • Definition 2.1: Average leave-one-out KL stability FS18
  • Definition 2.2: Average leave-many-out KL stability
  • Lemma 2.3: Subsampling queries are ALMOKL stable
  • ...and 89 more