Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Assumption-lean weak limits and tests for two-stage adaptive experiments

Ziang Niu, Zhimei Ren

TL;DR

This work develops assumption-lean weak convergence theory for a broad class of weighted IPW estimators in two-stage adaptive experiments, enabling valid inference under non-normal limiting distributions. It introduces a fast, plug-in bootstrap that leverages the derived weak limits to obtain accurate critical values without strong distributional assumptions, making hypothesis testing practical for adaptive designs. The framework unifies and extends prior results, showing phase transitions in limiting behavior across signal regimes and applying to batched bandits and subgroup enrichment, with extensive simulations and semi-synthetic data analyses demonstrating robust finite-sample performance. The approach facilitates design and power calculations for adaptive experiments, offering scalable, distribution-agnostic inference and providing a foundation for future extensions such as covariate adjustment and stopping-time adaptations.

Abstract

Adaptive experiments are becoming increasingly popular in real-world applications for effectively maximizing in-sample welfare and efficiency by data-driven sampling. Despite their growing prevalence, however, the statistical foundations for valid inference in such settings remain underdeveloped. Focusing on two-stage adaptive experimental designs, we address this gap by deriving new weak convergence results for mean outcomes and their differences. In particular, our results apply to a broad class of estimators, the weighted inverse probability weighted (WIPW) estimators. In contrast to prior works, our results require significantly weaker assumptions and sharply characterize phase transitions in limiting behavior across different signal regimes. Through this common lens, our general results unify previously fragmented results under the two-stage setup. To address the challenge of potential non-normal limits in conducting inference, we propose a computationally efficient and provably valid plug-in bootstrap method for hypothesis testing. Our results and approaches are sufficiently general to accommodate various adaptive experimental designs, including batched bandit and subgroup enrichment experiments. Simulations and semi-synthetic studies demonstrate the practical value of our approach, revealing statistical phenomena unique to adaptive experiments.

Assumption-lean weak limits and tests for two-stage adaptive experiments

TL;DR

This work develops assumption-lean weak convergence theory for a broad class of weighted IPW estimators in two-stage adaptive experiments, enabling valid inference under non-normal limiting distributions. It introduces a fast, plug-in bootstrap that leverages the derived weak limits to obtain accurate critical values without strong distributional assumptions, making hypothesis testing practical for adaptive designs. The framework unifies and extends prior results, showing phase transitions in limiting behavior across signal regimes and applying to batched bandits and subgroup enrichment, with extensive simulations and semi-synthetic data analyses demonstrating robust finite-sample performance. The approach facilitates design and power calculations for adaptive experiments, offering scalable, distribution-agnostic inference and providing a foundation for future extensions such as covariate adjustment and stopping-time adaptations.

Abstract

Adaptive experiments are becoming increasingly popular in real-world applications for effectively maximizing in-sample welfare and efficiency by data-driven sampling. Despite their growing prevalence, however, the statistical foundations for valid inference in such settings remain underdeveloped. Focusing on two-stage adaptive experimental designs, we address this gap by deriving new weak convergence results for mean outcomes and their differences. In particular, our results apply to a broad class of estimators, the weighted inverse probability weighted (WIPW) estimators. In contrast to prior works, our results require significantly weaker assumptions and sharply characterize phase transitions in limiting behavior across different signal regimes. Through this common lens, our general results unify previously fragmented results under the two-stage setup. To address the challenge of potential non-normal limits in conducting inference, we propose a computationally efficient and provably valid plug-in bootstrap method for hypothesis testing. Our results and approaches are sufficiently general to accommodate various adaptive experimental designs, including batched bandit and subgroup enrichment experiments. Simulations and semi-synthetic studies demonstrate the practical value of our approach, revealing statistical phenomena unique to adaptive experiments.
Paper Structure (138 sections, 32 theorems, 201 equations, 10 figures, 1 table, 2 algorithms)

This paper contains 138 sections, 32 theorems, 201 equations, 10 figures, 1 table, 2 algorithms.

Table of Contents

  1. Introduction
  2. Relevant literature
  3. Our contributions
  4. Organization of the paper
  5. Data generating procedure and test statistic
  6. Two-stage adaptive data collection
  7. Weighted IPW test statistic
  8. A broad class of weighting choices.
  9. Two scaling schemes.
  10. A peek at the sampling distribution.
  11. Theoretical results
  12. General theory: signal-dependent weak limits
  13. Form of limiting distributions.
  14. Remarks on the assumptions and technical challenges
  15. Phase transition and implication on hypothesis testing
  16. ...and 123 more sections

Key Result

Theorem 1

Suppose Assumptions assu:moment_condition-assu:sampling_design hold. Recall the definition $c_N\equiv \lim_{N\rightarrow\infty}\sqrt{N}(\mathbb E[Y_{uN}(0)]-\mathbb E[Y_{uN}(1)])$. The following statements hold.

Figures (10)

  • Figure 1: Sampling distribution of $\sqrt{N} T_N - c_N$ with adaptive weighting ($m = 1/2$).
  • Figure 2: Distribution $\mathbb{W}_{\mathcal{U}}^{\mathcal{A}}(c)$ as a function of limiting signal strength $c$.
  • Figure 3: QQ plots for the $5$ tests under different signal strengths. The simulation is repeated for $2000$ times. The number of bootstrap used in each test is $5000$.
  • Figure 4: Rejection rate for the $5$ tests under different signal strength. The simulation is repeated for $2000$ times. The number of bootstrap used in each test is $5000$.
  • Figure 5: Type-I error and power for the five tests under semi-synthetic data.
  • ...and 5 more figures

Theorems & Definitions (61)

  • Remark 1: Generality of $\mathcal{S}$
  • Theorem 1: Weak convergence
  • Remark 2: Comments on Assumptions \ref{['assu:moment_condition']}-\ref{['assu:adaptive_weighting']}
  • Remark 3: Early-dropping experiments
  • Remark 4: Technical challenges behind Theorem \ref{['thm:weak_convergence_W_N']}
  • Theorem 2: Smooth transition of limiting distributions
  • Remark 5: Generalization of Theorem \ref{['thm:weak_convergence_W_N']} to accommodate nuisance parameter
  • Remark 6: Computational efficiency
  • Remark 7: A nonparametric simulation-based approach
  • Theorem 3: Validity of plug-in bootstrap and tests $\hat{\phi}_{\mathcal{V}}^{\mathcal{W}}$
  • ...and 51 more