Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Fast Algorithms for Exact Confidence Intervals in Randomized Experiments with Binary Outcomes

Peng Zhang

TL;DR

Under a balanced Bernoulli design or a matched-pairs design, it is shown that exact confidence intervals can be computed using only $O(\log n)$ randomization tests, yielding an exponential reduction in the number of tests compared to brute-force, and an information-theoretic lower bound is proved.

Abstract

We construct exact confidence intervals for the average treatment effect in randomized experiments with binary outcomes using sequences of randomization tests. Our approach does not rely on large-sample approximations and is valid for all sample sizes. Under a balanced Bernoulli design or a matched-pairs design, we show that exact confidence intervals can be computed using only $O(\log n)$ randomization tests, yielding an exponential reduction in the number of tests compared to brute-force. We further prove an information-theoretic lower bound showing that this rate is optimal. In contrast, under balanced complete randomization, the most efficient known procedures require $O(n\log n)$ randomization tests (Aronow et al., 2023), establishing a sharp separation between these designs. In addition, we extend our algorithm to general Bernoulli designs using $O(n^2)$ randomization tests.

Fast Algorithms for Exact Confidence Intervals in Randomized Experiments with Binary Outcomes

TL;DR

Under a balanced Bernoulli design or a matched-pairs design, it is shown that exact confidence intervals can be computed using only randomization tests, yielding an exponential reduction in the number of tests compared to brute-force, and an information-theoretic lower bound is proved.

Abstract

We construct exact confidence intervals for the average treatment effect in randomized experiments with binary outcomes using sequences of randomization tests. Our approach does not rely on large-sample approximations and is valid for all sample sizes. Under a balanced Bernoulli design or a matched-pairs design, we show that exact confidence intervals can be computed using only randomization tests, yielding an exponential reduction in the number of tests compared to brute-force. We further prove an information-theoretic lower bound showing that this rate is optimal. In contrast, under balanced complete randomization, the most efficient known procedures require randomization tests (Aronow et al., 2023), establishing a sharp separation between these designs. In addition, we extend our algorithm to general Bernoulli designs using randomization tests.
Paper Structure (32 sections, 20 theorems, 118 equations, 4 tables, 4 algorithms)

This paper contains 32 sections, 20 theorems, 118 equations, 4 tables, 4 algorithms.

Table of Contents

  1. Introduction
  2. Problem Setup and Background
  3. Assignment Mechanism
  4. Exact Confidence Interval Construction by Inverting Randomization Tests
  5. Computational Challenge and Previous Works
  6. Our Results
  7. Roadmap.
  8. Binary Search, Lower Bounds, and Convergence Rates
  9. Computing $p_{\max}(\tau_0)$ Using at Most Two Randomization Tests for Balanced Bernoulli
  10. Monotonicity Properties of the Centered HT Estimator
  11. Unimodality of $S$
  12. Monotonicity of $p(a,b; \Delta)$ for $b \ge 1$.
  13. Monotonicity of $p(a,b; \Delta)$ for $b = 0$
  14. Computing the Maximum $p$-Value $p_{\max}(\tau_0)$
  15. Computing $p(a,b; \Delta)$ by the Fast Fourier Transform
  16. ...and 17 more sections

Key Result

Theorem 1.1

For any $\alpha \in (0,1)$ and observed data $(\boldsymbol{\mathit{Y}}^{\mathrm{obs}}, \boldsymbol{\mathit{Z}}^{\mathrm{obs}})$, under the balanced Bernoulli design or the matched-pairs design, the confidence set $\mathcal{I}_\alpha(\boldsymbol{\mathit{Y}}^{\mathrm{obs}}, \boldsymbol{\mathit{Z}}^{\m

Theorems & Definitions (33)

  • Theorem 1.1
  • Theorem 1.2
  • Theorem 1.3
  • Theorem 1.4
  • Lemma 2.1
  • Corollary 2.2
  • Lemma 2.3
  • Proposition 2.4
  • Lemma 3.1
  • proof
  • ...and 23 more