Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

On Nonasymptotic Confidence Intervals for Treatment Effects in Randomized Experiments

Ricardo J. Sandoval, Sivaraman Balakrishnan, Avi Feller, Michael I. Jordan, Ian Waudby-Smith

TL;DR

This work delivers nonasymptotic confidence intervals for the average treatment effect in randomized experiments that match the asymptotic effective sample size of $n\pi$. By introducing mini-batch complete randomization (MBCR) and leveraging negative dependence and variance adaptivity, the authors close the gap to CLT-based intervals, achieving widths scaling as $O(1/\sqrt{n\pi})$ and proving this rate is information-theoretically unimprovable. They provide Hoeffding-style, sub-Bernoulli, and variance-adaptive (cross-fitted) intervals, with rigorous analysis under both Bernoulli and complete randomization. An information-theoretic lower bound establishes the optimality of the $1/\sqrt{n\pi}$ scaling, and the MBCR construction yields unbiased estimators with favorable concentration properties. The results have practical implications for designing and analyzing finite-sample randomized experiments, including online and clustered settings, where tight finite-sample guarantees are crucial.

Abstract

We study nonasymptotic (finite-sample) confidence intervals for treatment effects in randomized experiments. In the existing literature, the effective sample sizes of nonasymptotic confidence intervals tend to be looser than the corresponding central-limit-theorem-based confidence intervals by a factor depending on the square root of the propensity score. We show that this performance gap can be closed, designing nonasymptotic confidence intervals that have the same effective sample size as their asymptotic counterparts. Our approach involves systematic exploitation of negative dependence or variance adaptivity (or both). We also show that the nonasymptotic rates that we achieve are unimprovable in an information-theoretic sense.

On Nonasymptotic Confidence Intervals for Treatment Effects in Randomized Experiments

TL;DR

This work delivers nonasymptotic confidence intervals for the average treatment effect in randomized experiments that match the asymptotic effective sample size of . By introducing mini-batch complete randomization (MBCR) and leveraging negative dependence and variance adaptivity, the authors close the gap to CLT-based intervals, achieving widths scaling as and proving this rate is information-theoretically unimprovable. They provide Hoeffding-style, sub-Bernoulli, and variance-adaptive (cross-fitted) intervals, with rigorous analysis under both Bernoulli and complete randomization. An information-theoretic lower bound establishes the optimality of the scaling, and the MBCR construction yields unbiased estimators with favorable concentration properties. The results have practical implications for designing and analyzing finite-sample randomized experiments, including online and clustered settings, where tight finite-sample guarantees are crucial.

Abstract

We study nonasymptotic (finite-sample) confidence intervals for treatment effects in randomized experiments. In the existing literature, the effective sample sizes of nonasymptotic confidence intervals tend to be looser than the corresponding central-limit-theorem-based confidence intervals by a factor depending on the square root of the propensity score. We show that this performance gap can be closed, designing nonasymptotic confidence intervals that have the same effective sample size as their asymptotic counterparts. Our approach involves systematic exploitation of negative dependence or variance adaptivity (or both). We also show that the nonasymptotic rates that we achieve are unimprovable in an information-theoretic sense.
Paper Structure (37 sections, 15 theorems, 189 equations, 6 figures, 1 algorithm)

This paper contains 37 sections, 15 theorems, 189 equations, 6 figures, 1 algorithm.

Table of Contents

  1. Introduction
  2. Preliminaries
  3. Canonical randomization procedures
  4. Mini-batch complete randomization
  5. Confidence Intervals with the Optimal Effective Sample Size
  6. Hoeffding-style and sub-Bernoulli confidence intervals
  7. Studentized variance-adaptive confidence intervals
  8. Information-Theoretic Lower Bounds
  9. Discussion
  10. Mini-batch complete randomization algorithm
  11. Proofs of confidence intervals
  12. Hoeffding-style confidence intervals
  13. Step 1: Exploiting between-group independence to write an expectation as a product.
  14. Step 2: Exploiting within-group dependence to apply Hoeffding's inequality.
  15. Step 3: Inverting Markov's inequality, optimizing the width, and union bounding.
  16. ...and 22 more sections

Key Result

Proposition 2.6

Let $\mathbb P_{\emph{MBCR}}$ and $\mathbb P_{\emph{CR}}$ be the probability distributions induced by mini-batch complete randomization and complete randomization, respectively. Then, for any permutation $(z_1, \dots, z_n)$ of $n_1$ ones and $n_0$ zeros,

Figures (6)

  • Figure 1: $\pi = 0.1$
  • Figure 2: $\pi = 0.01$
  • Figure 3: $\pi = 0.001$
  • Figure 5: The potential outcomes were generated as $Y_i(0) \sim \mathrm{Unif}(0.1, 0.5)$ and $Y_i(1) \coloneqq Y_i(0) + 0.5$.
  • Figure 6: The potential outcomes were generated as $Y_i(0) \sim \mathrm{Unif}(0.9, 1)$ and $Y_i(1) \coloneqq Y_i(0)$.
  • ...and 1 more figures

Theorems & Definitions (41)

  • Definition 2.4: Bernoulli randomization
  • Definition 2.5: Complete randomization
  • Proposition 2.6
  • Remark 2.7: Connection to stratified randomized experiments
  • Definition 2.8: Horvitz-Thompson estimator under mini-batch complete randomization
  • Proposition 2.9
  • Theorem 3.1
  • Corollary 3.2
  • proof : Proof sketch for \ref{['thm:hoeffding-mbcr']} and \ref{['corollary:scaling-hoeffding-mbcr']}
  • Remark 3.3
  • ...and 31 more