Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Stronger Neyman Regret Guarantees for Adaptive Experimental Design

Georgy Noarov, Riccardo Fogliato, Martin Bertran, Aaron Roth

TL;DR

This work tackles efficient adaptive sequential experimentation for unbiased ATE estimation in a finite-population design-based framework by introducing Neyman regret as a central performance metric. It develops ClipOGD$^{\mathrm{SC}}$, an enhanced noncontextual adaptive design that achieves $\widetilde{O}(\log T)$ Neyman regret through strong convexity and cost-sensitive clipping, and provides valid confidence intervals for the adaptive IPW estimator. Extending to contextual settings, the authors propose MGATE, a multigroup adaptive design that attains $\widetilde{O}(\sqrt{T})$ multigroup Neyman regret across potentially overlapping groups using a sleeping-experts meta-design. Empirical results on synthetic data, microfinance, and LLM benchmarking corroborate the theoretical gains, demonstrating faster convergence to Neyman-optimal treatment probabilities and lower group-specific regret, with practical validity for ATE and GATE-style inference in adaptive experiments.

Abstract

We study the design of adaptive, sequential experiments for unbiased average treatment effect (ATE) estimation in the design-based potential outcomes setting. Our goal is to develop adaptive designs offering sublinear Neyman regret, meaning their efficiency must approach that of the hindsight-optimal nonadaptive design. Recent work [Dai et al, 2023] introduced ClipOGD, the first method achieving $\widetilde{O}(\sqrt{T})$ expected Neyman regret under mild conditions. In this work, we propose adaptive designs with substantially stronger Neyman regret guarantees. In particular, we modify ClipOGD to obtain anytime $\widetilde{O}(\log T)$ Neyman regret under natural boundedness assumptions. Further, in the setting where experimental units have pre-treatment covariates, we introduce and study a class of contextual "multigroup" Neyman regret guarantees: Given any set of possibly overlapping groups based on the covariates, the adaptive design outperforms each group's best non-adaptive designs. In particular, we develop a contextual adaptive design with $\widetilde{O}(\sqrt{T})$ anytime multigroup Neyman regret. We empirically validate the proposed designs through an array of experiments.

Stronger Neyman Regret Guarantees for Adaptive Experimental Design

TL;DR

This work tackles efficient adaptive sequential experimentation for unbiased ATE estimation in a finite-population design-based framework by introducing Neyman regret as a central performance metric. It develops ClipOGD, an enhanced noncontextual adaptive design that achieves Neyman regret through strong convexity and cost-sensitive clipping, and provides valid confidence intervals for the adaptive IPW estimator. Extending to contextual settings, the authors propose MGATE, a multigroup adaptive design that attains multigroup Neyman regret across potentially overlapping groups using a sleeping-experts meta-design. Empirical results on synthetic data, microfinance, and LLM benchmarking corroborate the theoretical gains, demonstrating faster convergence to Neyman-optimal treatment probabilities and lower group-specific regret, with practical validity for ATE and GATE-style inference in adaptive experiments.

Abstract

We study the design of adaptive, sequential experiments for unbiased average treatment effect (ATE) estimation in the design-based potential outcomes setting. Our goal is to develop adaptive designs offering sublinear Neyman regret, meaning their efficiency must approach that of the hindsight-optimal nonadaptive design. Recent work [Dai et al, 2023] introduced ClipOGD, the first method achieving expected Neyman regret under mild conditions. In this work, we propose adaptive designs with substantially stronger Neyman regret guarantees. In particular, we modify ClipOGD to obtain anytime Neyman regret under natural boundedness assumptions. Further, in the setting where experimental units have pre-treatment covariates, we introduce and study a class of contextual "multigroup" Neyman regret guarantees: Given any set of possibly overlapping groups based on the covariates, the adaptive design outperforms each group's best non-adaptive designs. In particular, we develop a contextual adaptive design with anytime multigroup Neyman regret. We empirically validate the proposed designs through an array of experiments.

Paper Structure

This paper contains 51 sections, 9 theorems, 46 equations, 7 figures, 7 algorithms.

Table of Contents

  1. Introduction
  2. Our contributions
  3. Contribution I: Exponentially improved noncontextual Neyman regret bound.
  4. Contribution II: Adaptive designs with contextual Neyman regret guarantees.
  5. Organization
  6. Preliminaries
  7. Setting
  8. Adaptive design
  9. Objective: minimize variance of ATE estimator
  10. Efficient Non-Contextual ATE Estimation
  11. Adaptive Design with Logarithmic Neyman Regret
  12. Meta-Design: ClipOGD
  13. ClipOGD$^\mathrm{0}$: A $\widetilde{O}(\sqrt{T})$ regret design
  14. ClipOGD$^\mathrm{SC}$ : Our $\widetilde{O}(\log T)$ regret design
  15. Convergence of Adaptive Treatment Probabilities
  16. ...and 36 more sections

Key Result

Theorem 3.2

Suppose ass:bounds is satisfied with $C$, $c$ the corresponding constants. Let $h: \mathbb{N}_+ \to \mathbb{R}_{> 0}$ be strictly increasing. Let ClipOGD$^\mathrm{SC}$ be the adaptive design that instantiates alg:strong with learning rate $\eta_t = 1/(2c^2t)$ and clipping rate $\delta_t = 1/h(t)$. Since $h$ can be chosen to grow arbitrarily slowly, we can get: $\mathop{\mathbb{E}}[\mathrm{RegVar

Figures (7)

  • Figure 1: Treatment probabilities and Neyman regret of ClipOGD on Gaussian data for different noise ($\sigma$) levels. As $\sigma$ increases, ClipOGD$^\textrm{SC}$ converges more slowly. Its regret remains high, and the treatment probabilities do not settle within the observed time horizon ($T\approx 50{,}000$). The black line in the treatment probabilities indicates the Neyman optimal probability.
  • Figure 2: Treatment probabilities and Neyman regret of ClipOGD on microfinance data for $T\approx 15{,}000$ rounds.
  • Figure 3: Group-conditional Neyman regret of ClipOGD and MGATE on microfinance data. MGATE produces the lowest $\mathcal{G}$-multigroup Neyman regret as desired, and in this case dominates the non-contextual ClipOGD variants for each group, including the noncontextual group $G_0=\mathcal{X}$.
  • Figure 4: Treatment probabilities, variance of the ATE, and Neyman regret of ClipOGD on LLM benchmarking data. The solid black line in the treatment probabilities indicates the Neyman optimal probability.
  • Figure 5: Group-conditional Neyman regret of ClipOGD and MGATE on the LLM Benchmarking data.
  • ...and 2 more figures

Theorems & Definitions (29)

  • Definition 2.1: ATE
  • Definition 2.2: Adaptive IPW Estimator
  • Definition 2.3: Neyman Regret kato2020efficientdai2023clip
  • Theorem 3.2: Stronger Neyman Regret Bound
  • Lemma 3.3: $L_2$-Deviation from Benchmark Design
  • Corollary 3.4: $L_2$-Convergence to Benchmark Design
  • Corollary 3.5: Convergence in the Superpopulation Setting
  • proof
  • Theorem 3.7: CIs for Clipped Adaptive Designs
  • Definition 4.1: $\mathcal{G}$-Multigroup Neyman Regret
  • ...and 19 more