Adaptive Data Analysis for Growing Data

TL;DR

This paper presents the first generalization bounds for adaptive analysis on dynamic data, allowing the analyst to adaptively schedule their queries conditioned on the current size of the data, in addition to previous queries and responses, and incorporates time-varying empirical accuracy bounds and mechanisms, allowing for tighter guarantees as data accumulates.

Abstract

Reuse of data in adaptive workflows poses challenges regarding overfitting and the statistical validity of results. Previous work has demonstrated that interacting with data via differentially private algorithms can mitigate overfitting, achieving worst-case generalization guarantees with asymptotically optimal data requirements. However, such past work assumes data is static and cannot accommodate situations where data grows over time. In this paper we address this gap, presenting the first generalization bounds for adaptive analysis on dynamic data. We allow the analyst to adaptively schedule their queries conditioned on the current size of the data, in addition to previous queries and responses. We also incorporate time-varying empirical accuracy bounds and mechanisms, allowing for tighter guarantees as data accumulates. In a batched query setting, the asymptotic data requirements of our bound grows with the square-root of the number of adaptive queries, matching prior works' improvement over data splitting for the static setting. We instantiate our bound for statistical queries with the clipped Gaussian mechanism, where it empirically outperforms baselines composed from static bounds.

Figures (7)

• Figure 1: Schematic of our new setting for adaptive data analysis on growing data. The dataset is of size $n_0$ when the analysis begins, and grows by one data point in each round. The analyst asks queries adaptively in each round based on past responses, and receives a response from the mechanism before selecting the next query. The framework reduces to the static data setting when $n = n_0$.
• Figure 2: Outline of results in sec:theory. Arrows indicate key dependencies, and dashed boxes indicate results that hold for a particular class of queries.
• Figure 3: Comparison of the number of adaptive statistical queries that can be answered with error tolerance $\alpha = 0.1$ and uniform coverage probability $1 - \beta = 0.95$ using a growing dataset with growth ratio $n / n_0 = 3$ in a batched query setting. The number of queries (vertical axis) is plotted as a function of the final dataset size $n$ (horizontal axis), bound (curve style) and the number of query batches $b$ (horizontal panel). The right-most panel ($b = 1$), represents the static baseline setting where the analyst forgoes intermediate responses and submits all queries only after the entire dataset of size $n$ has arrived.
• Figure 4: Comparison of the number of adaptive statistical queries that can be answered with error tolerance $\alpha = 0.1$ and uniform coverage probability $1 - \beta = 0.95$ using a growing dataset with fixed initial size $n_0 = 500\,000$ in a batched query setting. The number of queries (vertical axis) is plotted as a function of the final dataset size $n$ (horizontal axis), bound (curve style) and the number of query batches $b$ (horizontal panel). The right-most panel with $b = 1$ corresponds to the static data setting.
• Figure 5: Comparison of the error tolerance $\alpha$ (vertical axis) for $k = 10\,000$ adaptive statistical queries with uniform coverage probability $1 - \beta = 0.95$ using different ADA bounds (curve style). The queries are answered using a growing dataset with final size $n$ (horizontal axis) and growth ratio $n / n_0 = 3$, and are grouped into $b$ batches (horizontal panel). The right-most panel with $b = 1$ corresponds to the static data setting.

Theorems & Definitions (60)

• definition 2.1
• definition 2.2
• definition 2.3: jung2020new
• definition 2.4
• lemma 3.0
• theorem 3.1: PS transfer theorem
• lemma 3.1
• theorem 3.2
• lemma 3.2
• theorem 3.3